I am pleased to announce a new version of PyFilesystem (0.3), which is a Python module that provides a common interface to many kinds of filesystem. Basically it provides a way of working with files and directories that is exactly the same, regardless of how and where the file information is stored. Even if you don't plan on working with anything other than the files and directories on your hard-drive, PyFilesystem can simplify your code and reduce the potential of error.
PyFilesystem is a joint effort by myself and Ryan Kelly, who has created a number of new FS implementations such as Amazon S3 support and Secure FTP, and some pretty cool features such as FUSE support and Django storage integration.
As an example of how awesome this package is, take a look at the following 6 lines of code, which creates a ramdrive:
from fs.osfs import OSFS from fs.memoryfs import MemoryFS from fs.expose import fuse home_fs = OSFS('~/') home_fs.makedir('ramdrive', allow_recreate=True) fuse.mount(MemoryFS(), home_fs.getsyspath('ramdrive'))
If you run this, a directory called ramdrive will appear in your home folder, the contents of which are stored purely in memory.
I prepared a screencast that gives a quick demonstration of some features – because if a picture is worth a thousand words, this video must be worth fifteen thousand words a second:
PyFilesystem screencast from Will McGugan on Vimeo.
See the project page on google code for more information, including API docs. There are also a couple of blog posts that will give a some more context.
This release has reached a good level of stability and maturity. I'd like to invite as many Pythonistas as possible to check out this module and possibly contribute to the project.
17 Responses to "PyFilesystem 0.3 released"
Hello,
This is the first time I stumble upon this library. I am particularly interested by the S3FS.
It seems to me particularly interesting to archive files and directories to S3. Do you have any idea on how you would mirror a large directory structure ? By large I mean 20 Gb (larger than the ram)
indeed my question is does “fs.utils.copydir” is limited by the ram or not ?
Thank you
Yml, ‘copydir’ copies a chunk at a time, which defaults to 16K, so ram won't be an issue. It should be pretty straightforward to do what you describe… :-)
Does it work with windows OS? Online docs mention only Mac and Linux.
Rene, yes, it works on Windows. I'll update the docs.
This project looks fantastic - so many use cases. Can't wait to give it a run.
Any plan or suggestion on supporting hadoop HDFS?
Awesome! The standard library doesn't even have a safe file copy (just shutil, if I'm not mistaken) so this is very welcome.
Decster, no plans at the moment. That would be a nice addition though.
Does this in any way relate to the filesystems abstraction talked about at Europython last year?
http://eagain.net/talks/pythonic-fs-2009/ [eagain.net]
Because that is a really neat tool I've used a few times.
Is it a problem if the “filesystem” is completely read-only? An example of this would be the archives of games, which have custom file formats, which I would know how to read, but not always how to write (or writing might be a less interesting use case)?
Simon, they are unrelated, although they do serve a similar purpose.
Koen, read-only filesystems are supported. The implementation just needs to prohibit opening files in write mode and any operations which alter the directory structure.
One immediate need I can see is to avoid having to monkey patch my MEDIA_ROOT in Django unit tests to be /tmp/…
It would make things faster too I suspect.
Any plan or suggestion on supporting Google Storage?
http://code.google.com/intl/es/apis/storage/ [code.google.com]
The ramdrive example requires membership in the group “fuse”. Check your account with the command “groups”. If needed, you can join with the following command:
sudo gpasswd -a YOUR_USER_NAME fuse
Great stuff. It's going to be a huge timesaver for me. I needed something that would allow me to both integrate with Django and allow a little lower-level tinkering. PyFilesystem will make this almost easy.
The documentation for the lower-level tinkering is more than enough to get me started on that part, but fs.expose.django_storage usage documentation is a bit sparse (ok, nonexistant ;-). Is there an example (good or otherwise) of using PyFilesystem with Django that someone could point me to?
Thanks for making my life a little easier.
Joe
Joe, I believe you set the following values in settings.py
If you have any problems, try the discussion group:
http://groups.google.com/group/pyfilesystem-discussion [groups.google.com]
Will, congratulations!
This projects rocks out loud!
I ever wanted a FS abstraction in python, it is REALLY useful for so many things.
One example, when using test-driven development, it really sucks to monkey patch os and shutil.
It can look silly, but is a awesome usage for a FS abstraction layer …
Thanks a lot!