June 20, 2010 will

PyFilesystem 0.3 released

I am pleased to announce a new version of PyFilesystem (0.3), which is a Python module that provides a common interface to many kinds of filesystem. Basically it provides a way of working with files and directories that is exactly the same, regardless of how and where the file information is stored. Even if you don't plan on working with anything other than the files and directories on your hard-drive, PyFilesystem can simplify your code and reduce the potential of error.

PyFilesystem is a joint effort by myself and Ryan Kelly, who has created a number of new FS implementations such as Amazon S3 support and Secure FTP, and some pretty cool features such as FUSE support and Django storage integration.

As an example of how awesome this package is, take a look at the following 6 lines of code, which creates a ramdrive:

from fs.osfs import OSFS
from fs.memoryfs import MemoryFS
from fs.expose import fuse

home_fs = OSFS('~/')
home_fs.makedir('ramdrive', allow_recreate=True)
fuse.mount(MemoryFS(), home_fs.getsyspath('ramdrive'))

If you run this, a directory called ramdrive will appear in your home folder, the contents of which are stored purely in memory.

I prepared a screencast that gives a quick demonstration of some features – because if a picture is worth a thousand words, this video must be worth fifteen thousand words a second:

PyFilesystem screencast from Will McGugan on Vimeo.

See the project page on google code for more information, including API docs. There are also a couple of blog posts that will give a some more context.

This release has reached a good level of stability and maturity. I'd like to invite as many Pythonistas as possible to check out this module and possibly contribute to the project.

Use Markdown for formatting
*Italic* **Bold** `inline code` Links to [Google](http://www.google.com) > This is a quote > ```python import this ```
your comment will be previewed here
gravatar
yml
Hello,
This is the first time I stumble upon this library. I am particularly interested by the S3FS.

It seems to me particularly interesting to archive files and directories to S3. Do you have any idea on how you would mirror a large directory structure ? By large I mean 20 Gb (larger than the ram)

indeed my question is does “fs.utils.copydir” is limited by the ram or not ?
Thank you
gravatar
Will McGugan
Yml, ‘copydir’ copies a chunk at a time, which defaults to 16K, so ram won't be an issue. It should be pretty straightforward to do what you describe… :-)
gravatar
Rene
Does it work with windows OS? Online docs mention only Mac and Linux.
gravatar
Will McGugan
Rene, yes, it works on Windows. I'll update the docs.
gravatar
BrettH
This project looks fantastic - so many use cases. Can't wait to give it a run.
gravatar
Decster
Any plan or suggestion on supporting hadoop HDFS?
gravatar
Pixy Misa
Awesome! The standard library doesn't even have a safe file copy (just shutil, if I'm not mistaken) so this is very welcome.
gravatar
Will McGugan
Decster, no plans at the moment. That would be a nice addition though.
gravatar
Simon Davy
Does this in any way relate to the filesystems abstraction talked about at Europython last year?

http://eagain.net/talks/pythonic-fs-2009/

Because that is a really neat tool I've used a few times.
gravatar
Koen
Is it a problem if the “filesystem” is completely read-only? An example of this would be the archives of games, which have custom file formats, which I would know how to read, but not always how to write (or writing might be a less interesting use case)?
gravatar
Will McGugan
Simon, they are unrelated, although they do serve a similar purpose.

Koen, read-only filesystems are supported. The implementation just needs to prohibit opening files in write mode and any operations which alter the directory structure.
gravatar
Peter Bengtsson
One immediate need I can see is to avoid having to monkey patch my MEDIA_ROOT in Django unit tests to be /tmp/…
It would make things faster too I suspect.
gravatar
twitio
Any plan or suggestion on supporting Google Storage?

http://code.google.com/intl/es/apis/storage/
gravatar
anonymous
The ramdrive example requires membership in the group “fuse”. Check your account with the command “groups”. If needed, you can join with the following command:

sudo gpasswd -a YOUR_USER_NAME fuse
gravatar
Joe
Great stuff. It's going to be a huge timesaver for me. I needed something that would allow me to both integrate with Django and allow a little lower-level tinkering. PyFilesystem will make this almost easy.

The documentation for the lower-level tinkering is more than enough to get me started on that part, but fs.expose.django_storage usage documentation is a bit sparse (ok, nonexistant ;-). Is there an example (good or otherwise) of using PyFilesystem with Django that someone could point me to?

Thanks for making my life a little easier.

Joe
gravatar
Will McGugan
Joe, I believe you set the following values in settings.py

 DEFAULT_FILE_STORAGE = fs.expose.django_storage.FSStorage
DEFAULT_FILE_STORAGE_FS = OSFS('foo/bar') # Or other FS

If you have any problems, try the discussion group:

http://groups.google.com/group/pyfilesystem-discussion
gravatar
Gabriel Falcão
Will, congratulations!

This projects rocks out loud!

I ever wanted a FS abstraction in python, it is REALLY useful for so many things.
One example, when using test-driven development, it really sucks to monkey patch os and shutil.
It can look silly, but is a awesome usage for a FS abstraction layer …

Thanks a lot!
gravatar
Mat
Great work, Will.
Are there any chances to run/execute files stored in memoryfs? Would it be technically possible?
gravatar
Will McGugan
Easiest way would be to copy it to the HD and run it there. Another alternative would be to use fuse. Otherwise you are limited to native apis.


Will
gravatar
Mat
Thanks for reply.
I have tested your ramdrive example with exposing fs.memoryfs to the os by fs.expose.fuse but it is impossible to set x permission to files under mount dir from os side.
What fuse alternative do you mean?

gravatar
Chris
Will,

Can I ask what books or methodologies you use to design software?

Thanks,


Chris
gravatar
Will
Chris, I'm not a big believer in methodologies when it comes to writing software. When I'm writing an interface that other developers may use, I put myself in their shoes and attempt to make it as easy to work with as possible. You can derive much from that simple rule of thumb.
gravatar
Julien
Hi

Great lib, thank you guys & happy new year :)

Just released a django+ExtJS explorer with pyfilesystem backend and drag&drop UI. https://github.com/revolunet/django-extjs-filebrowser

Works like a charm.

I miss you on github.