Where there is a Will

Sneak Preview of New Features in PyFilesystem 0.4

There have been some pretty exciting developments in PyFilesystem since version 0.3 was released – Ryan Kelly and myself have been hard at work, and there have been a number of excellent contributions to the code base from other developers. Version 0.4 will be released some time in January, but I'd like to give you a preview of some new features before the next version lands.

It is now possible to open any of the supported filesystems from a URL in this format, which makes it very easy to specify a filesystem (or individual file) from the command line or a config file. Here's a quick example that opens a bunch of quite different filesystems:

from fs.opener import fsopendir
projects_fs = fsopendir('/projects')
zip_fs = fsopendir('zip:///foo/bar/baz.zip')
ftp_fs = fsopendir('ftp://ftp/mozilla.org')
sftp_fs = fsopendir('sftp://example.org')

If you used Pyfilesystem in your application you could trivially change where your files are physically located, or where you save generated files to.

You can also open a file directly without the need to explicitly open the filesystem it is contained within, with the fsopen function, e.g.:

from fs.opener import fsopen
print fsopen('zip:///foo/bar/baz.zip!dir/somefile.txt').read()
print fsopen('ftp://ftp.mozilla.org/pub/README').read()

fsopen is very similar to the builtin open method and will return a file-like object of some kind. In fact, if you pass in a system path it works as open would (although the exceptions will be an instance of fs.errors.FSError rather than IOError).

Because of this similarity with the builtin, fsopen could be used to shadow open, and instantly add the ability for an application to open files on mediums other than a system drive. This is all it takes:

from fs.opener import fsopen as open

FS Commands

Version 0.4 also adds a number of applications that mirror some of the standard command line apps, but extend their functionality with FS URLs. For example, fsls, functions just like the ls command, but works with any of the supported filesystems:

will@will-linux:~$ fsls .
will@will-linux:~$ fsls zip://myzip
will@will-linux:~$ fsls ftp://ftp.mozilla.org/pub
will@will-linux:~$ fsls sftp://user:pass@securesite.com/home/will

You can also copy and move files between filesystems with fscp and fsmv, which work in a very similar manner to their cp and mv counterparts, with a few extensions such as multi-threaded copying (great for network filesystems) and an ascii progress bar. The following example copies all the .py files in my projects directory to zip file on an ftp server, and displays an progress bar to boot.

will@will-linux:~$ fscp ~/projects/*.py zip:ftp://will:password@example.org/backups/code.zip

Then there is the fscat command that writes a file in a filesystem to the terminal. The following example displays a python file in the zip file that we created with the previous command:

will@will-linux:~$ fscat zip:ftp://will:password@example.org/backups/code.zip!pythonrocks.py

The other commands fsmkdir, fsrm and fstree work as you may expect.

Serving

The fsserve command adds the ability to serve any of the supported filesystems over a number of protocols. The default is to serve http – in effect creating a webserver. The following is all it takes to serve the contents of your current working directory:

will@will-linux:~$ fsserve

Now if you point your browser at http://127.0.0.1 you will see a web-page with the contents of your current working directory (or a index.html file if present). It's not the most bullet-proof of web servers, but handy if you quickly want to serve some files. Naturally, fsserve works with any filesystem you pass to it. You could, for instance, serve the contents of a zip file without ever explicitly unzipping it, or create an ftp to http gateway by serving an ftp filesystem. The following command creates a ftp to http gateway for ftp.mozilla.org:

will@will-linux:~$ fsserve ftp://ftp.mozilla.org

You also have the option of serving a filesystem over SFTP (Secure FTP), or by RPC (Remote Procedure Call). Either of these two methods expose all the functionality of the remote filesystem, so you could run a server on one machine and create/move/copy/delete files from another machine on the network (or internet). For example, the following would serve the current working directory on localhost, port 3000:

will@will-linux:~$ fsserve -t rpc -p 3000

You can then connect to that server from another machine on the network. Assuming my local IP is 192.168.1.64 the following would display the directory structure from another machine on my network:

will@will-linux:~$ fstree rpc://192.168.1.64:3000

Mounting

Any of the filesystems can be mounted on the OS with the fsmount command, which uses FUSE on Linux or DOKAN on Windows. The advantage of this is that the filesystems exposed in Python can be used in any application, and browsed with Explorer or Nautilus. The following creates a ram drive on Linux:

will@will-linux:~$ fsmount mem:// mem

Or on Windows:

C:\> fsmount mem:// M

Get the Code

There is no documentation online for the new features as yet, but if you are a brave soul and want to experiment with any of the above commands then download the code from SVN and run python setup.py install. The command line apps all have a -h which which displays help on the various options.

Bear in mind that these commands are still somewhat experimental, and some of these commands have the capacity to delete files – so be careful. That said, I'm confident to use them for my day-to-day work.

Please see the projects page page if you want to report bugs or discuss Pyfilesystem with myself and the other developers.

Python 3 Object Oriented Programming

Packt Publising has sent me a copy of Python 3 Object Oriented Programming to review. It's quite a hefty tome and will take me a few days to get through. In the meantime, they have a free chapter to whet your appetite.

Python 3 Object Oriented Programming

I'm liking this trend of free books! Watch this space for the review…

What to do with Locidesktop?

So what to do with locidesktop.com? It's a desktop-like website bookmarking tool – if you haven't seen it, take a quick look at this example desktop.

I built Loci Desktop a few months ago and promoted it on a few geek sites. It's been running ever since, with no maintenance from myself, happily serving up start pages to a small number of regular users. There was a buzz when I promoted it, people were largely impressed, some were indifferent, but few ended up using it regularly. So now I'm left with a quandary.

I could try and promote it. But to what end? It's not like I need a certain number of visitors to cover the hosting. I'm using the same VPS as I am for my blog, and I designed Locidesktop to be ultra-low bandwidth anyway – so it effectively costs me nothing to run.

One option would be to sell the entire site outright, as the domain and technology rights. But there is currently no way of monetizing it and I doubt anyone would be interested as a commercial venture.

I could try and license it as a b2b service. A few people have commented that it would be a useful intranet service. I'm not sure about this, but it sounds plausible.

Alternatively, I could give back to the Django community and release it as open source, which I imagine would be the popular thing to do. Money isn't my primary motivator (a close second perhaps), so I wouldn't be averse to doing this. Thing is though, it would require work on my part to document it and maintain it, and I have other open source projects I would prefer to concentrate on. But I can't deny that it would be cool to see Locidesktop sites popping up over the interwebs.

Finally, I could just leave it as is. I'm pleased with how it turned out, and I have a few loyal users. Maybe I should just be satisfied.

Any options I haven't considered? Leave a comment…

PyFilesystem 0.3 released

I am pleased to announce a new version of PyFilesystem (0.3), which is a Python module that provides a common interface to many kinds of filesystem. Basically it provides a way of working with files and directories that is exactly the same, regardless of how and where the file information is stored. Even if you don't plan on working with anything other than the files and directories on your hard-drive, PyFilesystem can simplify your code and reduce the potential of error.

PyFilesystem is a joint effort by myself and Ryan Kelly, who has created a number of new FS implementations such as Amazon S3 support and Secure FTP, and some pretty cool features such as FUSE support and Django storage integration.

As an example of how awesome this package is, take a look at the following 6 lines of code, which creates a ramdrive:

from fs.osfs import OSFS
from fs.memoryfs import MemoryFS
from fs.expose import fuse

home_fs = OSFS('~/')
home_fs.makedir('ramdrive', allow_recreate=True)
fuse.mount(MemoryFS(), home_fs.getsyspath('ramdrive'))

If you run this, a directory called ramdrive will appear in your home folder, the contents of which are stored purely in memory.

I prepared a screencast that gives a quick demonstration of some features – because if a picture is worth a thousand words, this video must be worth fifteen thousand words a second:

PyFilesystem screencast from Will McGugan on Vimeo.

See the project page on google code for more information, including API docs. There are also a couple of blog posts that will give a some more context.

This release has reached a good level of stability and maturity. I'd like to invite as many Pythonistas as possible to check out this module and possibly contribute to the project.

Review of Django 1.2 E-commerce

I've worked with Django for more than two years now. The majority of the sites I have worked on have been social-networking or content based, but I have yet to do any serious work on a site where the main purpose is to advertise and sell products. So I when a copy of ‘Django 1.2 e-commerce’ landed on my desk I was intrigued by what it might cover that I hadn't been exposed to with other fields of Django development.

Django 1.2 E-commerce

The book starts out with a brief run-down of Django. The first chapter is more of a explanation of the philosophy behind Django, and definitely not a tutorial. Which I think is fair enough; if you are building an e-commerce site, you are probably a professional Python developer and there are plenty of books to get you up to speed with Django. In the second chapter, the author runs through a simple web-shop application with an inventory and a ‘buy now’ button – which seemed more like a confidence building exercise than anything else, but it does do a good job of demonstrating how simple it can be to build this kind of application with Django.

The subsequent chapters go in to detail regarding managing users, shopping-carts and taking payments with Google Checkout and Amazon services. I've never used these payment services, so I found the information particularly interesting. I like the author's approach of making generic views in order to share functionality across payment services, but I would have liked some more detail in to the APIs involved.

Chapter 6 was a surprise, it covers a variety of modules that you can use to add powerful search capabilities to your application. Django's database querying will only get you so far with searching, if you need more sophisticated searching of the kind you would expect from Google then you will need to integrate one of a number of external modules and services, which chapter 6 covers pretty well. The following chapter covers exposing data via several APIs, and creating PDF reports with ReportLab, but not in any great detail.

I was impressed with chapter 8, which covers writing JavaScript to create rich AJAX interfaces – something which is pretty much expected in a modern web-site. I would have preferred a more detail here, but only because I have a particular interest in front-end technologies. The next chapter explains how to integrate a Django application with Amazon payment services and S3 storage to sell digital goods, and goes it to more detail than other chapters.

The final chapter covers a number of options you will have for deploying your application. If your application is moderately sophisticated and has many components, deployment can be a tricky affair. This chapter explains how Python technologies such as Fabric, Buildout and Virtualenv can ease deployment headaches. It also covers serving the site with Apache and ‘mod_wsgi’.

The code snippets in this book are pretty good at demonstrating the subjects covered, but I did notice some quite glaring syntax errors in several of the code examples. The errors weren't subtle either; they would result in the code not even running. I suspect that many of them were likely to be the result of a non-technical editor re-formating the code and not a mistake on the part of the author, but there were also a number of programming errors and bad practices which were a little disapointing to see in a book aimed at professionals. For instance, the author consistently used the ‘is’ operator in place of the equality operator (they are not interchangable even though they may appear to be).

Overall, my impression of this book was favourable. It's definitely not a tutorial book in that its not going to teach you any new skills – since it covers so many technologies and doesn't go in to great detail about any of them. What it will do is give you a grounding of the components in an e-commerce system. If you are looking to build some kind of web-shop in Django then I would recommend this book. It's less of an essential purchase if you aren't working with e-commerce, but since many of the topics discussed in Django 1.2 e-commerce are applicable to other web-sites you may still want to check this book out.

Creationists in Oxford

I often read about creationism in the US, and its followers. I tend to find it amusing, and just a little worrying as their numbers are not insignificant. We have plenty of street preachers here in the UK, but until today I have never seen those US-style anti-science evolution deniers on the streets.

Creationists

I watched for 10 minutes as I ate my sausage roll, and listened to the standard creationist spiel of how everything must have a beginning and an end, and how everything created must have a creator and how there are no transitional fossils. It was quite entertaining!

Amazing to have that right here in Oxford, a place of learning. There's a natural history museum just around the corner. I wonder if they had ever been?

Announcing Sore Thumb, a thumbnail and image processing module for Django

I recently worked on the re-design of 2 Degrees, which required a lot of image processing on thumbnails. The thumbnails where to be in a variety of different sizes, all with rounded corners and keylines on a selection of virtually identical off-white backgrounds and gradients. And they all had to work on IE6 *spit* without the transparency hack.

A variety of thumbnails generated by Sore Thumb

A lesser engineer may have told the front-end developer where to stick his rounded corners, but I didn't want see a grown man cry, so I built Sore Thumb, an on-the-fly thumbnail and image processing system for Django.

Sore Thumb uses a declarative method of defining thumbnails, similar to Django's model and form definitions. Here's an example of how to declare a thumbnail processor that produces a 120x100 pixel thumbnail with 10 pixel rounded corners and a dark grey keyline:

from sorethumb.djangothumbnail import DjangoThumbnail
from sorethumb.filters.defaultfilters import ThumbnailFilter
from sorethumb.filters.drawfilters import RoundedCornerFilter

class RoundedCornersEdged(DjangoThumbnail):
    format = 'png'
    filters = [ThumbnailFilter(120, 100),
               RoundedCornerFilter(10, border='#333')]

Once this class has been imported, the thumbnail processor will be available in templates via the sorethumb filter which takes a Django FileField and returns the url to the thumbnail.

For example:

{% load sorethumb %}

<img src="{{ profile.photo|sorethumb:"rounded_corners_edged" }} />

You will also need to add sorethumb to your INSTALLED_APPS for this to work.

That's pretty much all there is to working with Sore Thumb, see these examples for inspiration and the documentation for the details.

You can install sorethumb with easy_install, PIP, or directly from source:

easy_install -U sorethumb

Please let me know what you think of Sore Thumb. I'd rather not spend too much time maintaining it, but since most of the code has been in production for a number of months already, there shouldn't be many changes required. If you have any feature requests or bug-reports, now would be a good time to raise them while I have the time to do the work!

Django E-Commerce

Pakt publishing have released a new Django book called Django 1.2 E-Commerce, written by Jess Legg. I'll have a copy soon and post a review here.

In the meantime they are offering a free chapter of the book; Chapter 2 Setting up Shop in 30 minutes.

Exciting Python Developer Job

Meebo, the instant messenger in your browser company, are seriously looking for Python developers right now to work in Mountain View, California. From what I can gather they are expanding, and are building a number of sites in the Django framework.

Agnostic or Atheist?

Alright. Here is my second attempt at VBlogging. I'm getting better at talking to the camera, which I find difficult to do. Still can't do it in one cut though!

It's on the subject of Agnosticism and Atheism. It's probably old territory for most people interested in this subject matter, but hopefully somebody will find it interesting…