There have been some pretty exciting developments in PyFilesystem since version 0.3 was released – Ryan Kelly and myself have been hard at work, and there have been a number of excellent contributions to the code base from other developers. Version 0.4 will be released some time in January, but I'd like to give you a preview of some new features before the next version lands.

Pyfilesystem is a Python module that provides a simplified common interface to many types of filesystem.

It is now possible to open any of the supported filesystems from a URL in this format, which makes it very easy to specify a filesystem (or individual file) from the command line or a config file. Here's a quick example that opens a bunch of quite different filesystems:

from fs.opener import fsopendir
projects_fs = fsopendir('/projects')
zip_fs = fsopendir('zip:///foo/bar/baz.zip')
ftp_fs = fsopendir('ftp://ftp/mozilla.org')
sftp_fs = fsopendir('sftp://example.org')

If you used Pyfilesystem in your application you could trivially change where your files are physically located, or where you save generated files to.

You can also open a file directly without the need to explicitly open the filesystem it is contained within, with the fsopen function, e.g.:

from fs.opener import fsopen
print fsopen('zip:///foo/bar/baz.zip!dir/somefile.txt').read()
print fsopen('ftp://ftp.mozilla.org/pub/README').read()

fsopen is very similar to the builtin open method and will return a file-like object of some kind. In fact, if you pass in a system path it works as open would (although the exceptions will be an instance of fs.errors.FSError rather than IOError).

Because of this similarity with the builtin, fsopen could be used to shadow open, and instantly add the ability for an application to open files on mediums other than a system drive. This is all it takes:

from fs.opener import fsopen as open

FS Commands

Version 0.4 also adds a number of applications that mirror some of the standard command line apps, but extend their functionality with FS URLs. For example, fsls, functions just like the ls command, but works with any of the supported filesystems:

will@will-linux:~$ fsls .
will@will-linux:~$ fsls zip://myzip
will@will-linux:~$ fsls ftp://ftp.mozilla.org/pub
will@will-linux:~$ fsls sftp://user:pass@securesite.com/home/will

You can also copy and move files between filesystems with fscp and fsmv, which work in a very similar manner to their cp and mv counterparts, with a few extensions such as multi-threaded copying (great for network filesystems) and an ascii progress bar. The following example copies all the .py files in my projects directory to zip file on an ftp server, and displays an progress bar to boot.

will@will-linux:~$ fscp ~/projects/*.py zip:ftp://will:password@example.org/backups/code.zip

Then there is the fscat command that writes a file in a filesystem to the terminal. The following example displays a python file in the zip file that we created with the previous command:

will@will-linux:~$ fscat zip:ftp://will:password@example.org/backups/code.zip!pythonrocks.py

The other commands fsmkdir, fsrm and fstree work as you may expect.

Serving

The fsserve command adds the ability to serve any of the supported filesystems over a number of protocols. The default is to serve http – in effect creating a webserver. The following is all it takes to serve the contents of your current working directory:

will@will-linux:~$ fsserve

Now if you point your browser at http://127.0.0.1 you will see a web-page with the contents of your current working directory (or a index.html file if present). It's not the most bullet-proof of web servers, but handy if you quickly want to serve some files. Naturally, fsserve works with any filesystem you pass to it. You could, for instance, serve the contents of a zip file without ever explicitly unzipping it, or create an ftp to http gateway by serving an ftp filesystem. The following command creates a ftp to http gateway for ftp.mozilla.org:

will@will-linux:~$ fsserve ftp://ftp.mozilla.org

You also have the option of serving a filesystem over SFTP (Secure FTP), or by RPC (Remote Procedure Call). Either of these two methods expose all the functionality of the remote filesystem, so you could run a server on one machine and create/move/copy/delete files from another machine on the network (or internet). For example, the following would serve the current working directory on localhost, port 3000:

will@will-linux:~$ fsserve -t rpc -p 3000

You can then connect to that server from another machine on the network. Assuming my local IP is 192.168.1.64 the following would display the directory structure from another machine on my network:

will@will-linux:~$ fstree rpc://192.168.1.64:3000

Mounting

Any of the filesystems can be mounted on the OS with the fsmount command, which uses FUSE on Linux or DOKAN on Windows. The advantage of this is that the filesystems exposed in Python can be used in any application, and browsed with Explorer or Nautilus. The following creates a ram drive on Linux:

will@will-linux:~$ fsmount mem:// mem

Or on Windows:

C:\> fsmount mem:// M

Get the Code

There is no documentation online for the new features as yet, but if you are a brave soul and want to experiment with any of the above commands then download the code from SVN and run python setup.py install. The command line apps all have a -h which which displays help on the various options.

Bear in mind that these commands are still somewhat experimental, and some of these commands have the capacity to delete files – so be careful. That said, I'm confident to use them for my day-to-day work.

Please see the projects page page if you want to report bugs or discuss Pyfilesystem with myself and the other developers.

This blog post was posted to It's All Geek to Me on Saturday January 1st, 2011 at 4:26PM
 

8 Responses to "Sneak Preview of New Features in PyFilesystem 0.4"

  • Stu
    January 1st, 2011, 8:24 p.m.

    Great stuff… is WebDAV included/coming?

    Can this be used as a library ?

  • January 1st, 2011, 8:27 p.m.

    Stu,

    WebDAV is supported now. Everything can be done programatically. The command line apps are just scripts that use Pyfilesystem.

  • Alia Khouri
    January 2nd, 2011, 3:36 p.m.

    Very nice. Can I suggest you provide a single unified interface to the command line commands:

    > fs cat

    > fs cp

    > fs rm

    > fs mount

    > fs serve

    etc..

    argparse (and extensions eg. argh) is nice for this kind of thing.

    AK

  • January 5th, 2011, 11:31 p.m.

    Hi

    Just implemented your greeeat lib in a django app for browsing differents filesystems right from the browser.

    Works like a charm, thank you so much guys.

    I plan to add google docs support.

    https://github.com/revolunet/django-extjs-filebrowser [github.com]

    And, if you dont mind, you should really move to github :)

  • May 16th, 2011, 12:09 p.m.

    Wow, this is absolutely great!

    I can't wait to see more features. I am mostly interested in a memory encrypted filesystem, for working with sensitive data.

    Great project!!

  • Max Spencer
    August 30th, 2011, 4:33 p.m.

    Hey,

    I am having some trouble mounting using Dokan. I just keep getting the following exception when I go to mount an fs via Dokan:

    >>> dokan.mount(testfs, ‘Q’)

    Traceback (most recent call last):

    File “<stdin>”, line 1, in <module>

    File “C:\python27\lib\site-packages\fs\expose\dokan\__init__.py”, line 887, in mount

    raise OSError(“the dokan library is not available”)

    OSError: the dokan library is not available

    I am a bit of a Python noob and I've not used Dokan before.

  • Merlink
    March 8th, 2012, 11:18 a.m.

    You have to install Dokan from here:

    http://dokan-dev.net/en/download/

  • daniel
    September 27th, 2012, 5:16 a.m.

    Excellent stuff!

    I have a trouble with fsmount.
    On Windows, ‘C:\> fsmount mem:// M’ will create a ram drive. This is no problem.

    The problem is when I copy a large file to the ram drive, bigger than the system memory size, it overflows and the system crashs. Do you have any way to prevent it, e.g. spool the overflow content to physical drive?

    Thank you.

Leave a Comment

You can use bbcode in the comment: e.g. [b]This is bold[/b], [url]http://www.willmcgugan.com[/url], [code python]import this[/code]
Preview Posting...
Previewing comment, please wait a moment...

My Tweets

Will McGugan

My name is Will McGugan. I am an unabashed geek, an author, a hacker and a Python expert – amongst other things!

Search for Posts
Possibly related posts
Tags
Popular Tags
 
Archives
2013
 
Recent Comments
Sir, can you give me full code.?
Trying to implement this code. What are the requirements? Does it need a certain version of PHP, etc? Do I ...
Hi Will Can't seem to run any pyopengl applications on my Win7 64bit setup. Using python 3.3 but keep getting ...
Great work, is it possible to jump to a specified picture?
Thanks for the lesson and library, but: What if I wanted to limit the frames as well? Is it really ...
- The novice on Master time with PyGame
 
© 2008 Will McGugan.

A technoblog blog, design by Will McGugan