I'd like to announce an new Python module to make working with Amazon S3 files a whole lot easier.

The S3FS class in fs-s3fs wraps an Amazon S3 bucket in a PyFilesystem interface. There was an S3FS class built in to the first version of PyFilesystem, but it had suffered from using an older version of 'boto' (Amazon's S3 interface) and was in need of maintenance. The new version is up-to-date with PyFilesystem2 and boto3, and works with Python2.7 and Python3.X.

If you aren't familiar with PyFilesystem, it is a common interface to anything that resembles a filesystem. Amazon S3 isn't quite a full filesystem, but close enough that it can be made to work in the same way as the files and directories on your local drive (or any other supported filesystem for that matter).

For instance here's how you might uploads files from your local drive to an S3 bucket:

from fs.copy import copy_dir
copy_dir('~/projects', 's3://backups')

This backs up a 'projects' folder in your home directory to a bucket called 'backups' with a simple file copy. The two strings are FS URLs and could refer to any of the supported filesystems. You could use the same function to download files from S3 straight in to a zip file with copy_dir('s3://backups', 'zip://bucket.zip') -- literally any combination of source and destination will work.

This magic is supplied by a relatively new feature to PyFilesystem, which is the automatic discovery of new protocols. The beauty of this system is that applications using PyFilesystem can now work with S3 without any changes. For example Moya gains the ability to serve index pages for S3 buckets.

If you have S3 configured on your system, here's how you can serve index pages for a bucket:

$ pip install moya fs-s3fs -U
$ moya serve s3://mybucket --show-debug

I've been guilty of using this blog for announcements rather than discussing techie things, but I kind of feel that the magic here is worthy of a more detailed blog post. Let me know in the comments if there is anything that could use further explanation.

If you have any issues with S3FS (it's a relatively young project), let me know on S3FS GitHub issues.