I am pleased to announce a new version of PyFilesystem (0.3), which is a Python module that provides a common interface to many kinds of filesystem. Basically it provides a way of working with files and directories that is exactly the same, regardless of how and where the file information is stored. Even if you don't plan on working with anything other than the files and directories on your hard-drive, PyFilesystem can simplify your code and reduce the potential of error.

PyFilesystem is a joint effort by myself and Ryan Kelly, who has created a number of new FS implementations such as Amazon S3 support and Secure FTP, and some pretty cool features such as FUSE support and Django storage integration.

As an example of how awesome this package is, take a look at the following 6 lines of code, which creates a ramdrive:

from fs.osfs import OSFS
from fs.memoryfs import MemoryFS
from fs.expose import fuse

home_fs = OSFS('~/')
home_fs.makedir('ramdrive', allow_recreate=True)
fuse.mount(MemoryFS(), home_fs.getsyspath('ramdrive'))

If you run this, a directory called ramdrive will appear in your home folder, the contents of which are stored purely in memory.

I prepared a screencast that gives a quick demonstration of some features – because if a picture is worth a thousand words, this video must be worth fifteen thousand words a second:

PyFilesystem screencast from Will McGugan on Vimeo.

See the project page on google code for more information, including API docs. There are also a couple of blog posts that will give a some more context.

This release has reached a good level of stability and maturity. I'd like to invite as many Pythonistas as possible to check out this module and possibly contribute to the project.

This blog post was posted to It's All Geek to Me on Sunday June 20th, 2010 at 5:02PM
 

23 Responses to "PyFilesystem 0.3 released"

  • yml
    June 20th, 2010, 8:06 p.m.

    Hello,

    This is the first time I stumble upon this library. I am particularly interested by the S3FS.

    It seems to me particularly interesting to archive files and directories to S3. Do you have any idea on how you would mirror a large directory structure ? By large I mean 20 Gb (larger than the ram)

    indeed my question is does “fs.utils.copydir” is limited by the ram or not ?

    Thank you

  • June 20th, 2010, 8:33 p.m.

    Yml, ‘copydir’ copies a chunk at a time, which defaults to 16K, so ram won't be an issue. It should be pretty straightforward to do what you describe… :-)

  • Rene
    June 20th, 2010, 10:42 p.m.

    Does it work with windows OS? Online docs mention only Mac and Linux.

  • June 20th, 2010, 10:44 p.m.

    Rene, yes, it works on Windows. I'll update the docs.

  • June 21st, 2010, 1:17 a.m.

    This project looks fantastic - so many use cases. Can't wait to give it a run.

  • Decster
    June 21st, 2010, 4:51 a.m.

    Any plan or suggestion on supporting hadoop HDFS?

  • June 21st, 2010, 7:55 a.m.

    Awesome! The standard library doesn't even have a safe file copy (just shutil, if I'm not mistaken) so this is very welcome.

  • June 21st, 2010, 8:53 a.m.

    Decster, no plans at the moment. That would be a nice addition though.

  • Simon Davy
    June 21st, 2010, 10:06 a.m.

    Does this in any way relate to the filesystems abstraction talked about at Europython last year?

    http://eagain.net/talks/pythonic-fs-2009/ [eagain.net]

    Because that is a really neat tool I've used a few times.

  • June 21st, 2010, 1:41 p.m.

    Is it a problem if the “filesystem” is completely read-only? An example of this would be the archives of games, which have custom file formats, which I would know how to read, but not always how to write (or writing might be a less interesting use case)?

  • June 21st, 2010, 1:53 p.m.

    Simon, they are unrelated, although they do serve a similar purpose.

    Koen, read-only filesystems are supported. The implementation just needs to prohibit opening files in write mode and any operations which alter the directory structure.

  • June 21st, 2010, 5:35 p.m.

    One immediate need I can see is to avoid having to monkey patch my MEDIA_ROOT in Django unit tests to be /tmp/…

    It would make things faster too I suspect.

  • twitio
    June 21st, 2010, 9:10 p.m.

    Any plan or suggestion on supporting Google Storage?

    http://code.google.com/intl/es/apis/storage/ [code.google.com]

  • anonymous
    June 23rd, 2010, 1:41 a.m.

    The ramdrive example requires membership in the group “fuse”. Check your account with the command “groups”. If needed, you can join with the following command:

    sudo gpasswd -a YOUR_USER_NAME fuse

  • Joe
    June 24th, 2010, 8:22 p.m.

    Great stuff. It's going to be a huge timesaver for me. I needed something that would allow me to both integrate with Django and allow a little lower-level tinkering. PyFilesystem will make this almost easy.

    The documentation for the lower-level tinkering is more than enough to get me started on that part, but fs.expose.django_storage usage documentation is a bit sparse (ok, nonexistant ;-). Is there an example (good or otherwise) of using PyFilesystem with Django that someone could point me to?

    Thanks for making my life a little easier.

    Joe

  • June 24th, 2010, 8:34 p.m.

    Joe, I believe you set the following values in settings.py

    DEFAULT_FILE_STORAGE = fs.expose.django_storage.FSStorage
    DEFAULT_FILE_STORAGE_FS = OSFS('foo/bar') # Or other FS

    If you have any problems, try the discussion group:

    http://groups.google.com/group/pyfilesystem-discussion [groups.google.com]

  • August 20th, 2010, 3:17 p.m.

    Will, congratulations!

    This projects rocks out loud!

    I ever wanted a FS abstraction in python, it is REALLY useful for so many things.

    One example, when using test-driven development, it really sucks to monkey patch os and shutil.

    It can look silly, but is a awesome usage for a FS abstraction layer …

    Thanks a lot!

  • Mat
    November 9th, 2010, 1:52 p.m.

    Great work, Will.

    Are there any chances to run/execute files stored in memoryfs? Would it be technically possible?

  • November 9th, 2010, 1:58 p.m.

    Easiest way would be to copy it to the HD and run it there. Another alternative would be to use fuse. Otherwise you are limited to native apis.

    Will

  • Mat
    November 9th, 2010, 2:23 p.m.

    Thanks for reply.

    I have tested your ramdrive example with exposing fs.memoryfs to the os by fs.expose.fuse but it is impossible to set x permission to files under mount dir from os side.

    What fuse alternative do you mean?

  • Chris
    November 17th, 2010, 4:23 a.m.

    Will,

    Can I ask what books or methodologies you use to design software?

    Thanks,

    Chris

  • November 17th, 2010, 2:34 p.m.

    Chris, I'm not a big believer in methodologies when it comes to writing software. When I'm writing an interface that other developers may use, I put myself in their shoes and attempt to make it as easy to work with as possible. You can derive much from that simple rule of thumb.

  • January 6th, 2011, 10:47 a.m.

    Hi

    Great lib, thank you guys & happy new year :)

    Just released a django+ExtJS explorer with pyfilesystem backend and drag&drop UI. https://github.com/revolunet/django-extjs-filebrowser [github.com]

    Works like a charm.

    I miss you on github.

Leave a Comment

You can use bbcode in the comment: e.g. [b]This is bold[/b], [url]http://www.willmcgugan.com[/url], [code python]import this[/code]
Preview Posting...
Previewing comment, please wait a moment...
Will McGugan

My name is Will McGugan. I am an unabashed geek, an author, a hacker and a Python expert – amongst other things!

You are reading my tech blog. See the homepage for my other blogs.

Search for Posts
Possibly related posts
Tags
Popular Tags
 
Archives
2013
 
Recent Comments
Beautiful. Just beautiful. Thank you.
- Tyler Troy on Going sub-pixel with PyGame
Sorry for the double comment my browser is very slow.
Hi Will I get the following error when i try to run simpleopengl.py. Traceback (most recent call last): File firstopengl.py, ...
Hi Will I get the following error when i try to run simpleopengl.py. Traceback (most recent call last): File firstopengl.py, ...
Men, you are awesome. You are so great. It works so fucking good and it was so easy to install. ...
 
© 2008 Will McGugan.

A technoblog blog, design by Will McGugan