Python Posts

166 posts tagged with python

Jan 13, 2019
in Tech

Filesystem Magic with Python

I recently added a number of examples on working with files and directories with Python and PyFilesystem.

The first example, is count_py.py which recursively sums up the number of bytes stored in a directory, and renders it in a "friendly" format.

This script takes a path or a URL. For instance python3 count_py.py ~/projects will sum up the bytes of Python in your projects folder.

For me, this gives the following output:

This script accepts a path, but it will also work with any supported filesystem. So you could sum the bytes of Python in an archive or cloud server. continue reading…

Oct 15, 2018
in Tech

Adding type hints to the Django ORM

It occurred to me that Django's ORM could do with a bit of a revamp to make use of recent developments in the Python language.

The main area where I think Django's models are missing out is the lack of type hinting (hardly surprising since Django pre-dates type hints). Adding type hints allows Mypy to detect bugs before you even run your code. It may only save you minutes each time, but multiply that by the number of code + run iterations you do each day, and it can save hours of development time. Multiply that by the lifetime of your project, and it could save weeks or months. A clear win. continue reading…

Aug 12, 2018
in Tech

PyFilesystem 2.1.0 adds concurrent uploads / downloads and support for globbing

I've released PyFilesystem 2.1.0.

This version is the accumulation of many minor revisions, with multiple fixes, enhancements and some interesting new features. We also have nicer doc strings and Mypy compatible typing information, thanks to Martin Larralde.

If you aren't familiar with PyFilesystem, it is a common Python API to filesystems, whether its your hard-drive, archives, memory, FTP servers, or cloud services. The API is easier to use than the standard library, but works just about everywhere. See the Pyfilesystem Wiki for more information. continue reading…

Jul 5, 2018
in Tech

Idiomatic usage of Python assignment expressions (PEP 572)

Most PEPs (Python Enhancement Proposal) tend do go under the radar of most developers, but PEP 572 has caused a lot of controversy in the Python community, with some developers expressing an intense dislike for the new syntax (and that is sugar coated).

Personally, I don't think it deserves the vitriol it received, and I'm looking forward to its addition in Python 3.8.

So what are assignment expressions? Currently in Python assignments have to be statements, so you can do foo = bar(), but you can't do that assignment from within an if or while statement (for example). Which is why the following is a syntax error: continue reading…

Jul 5, 2018
in Tech

Compress WebSocket streams with Lomond 0.3.2

I've recently released version 0.3.2 of Lomond, a WebSocket client for Python with a focus on correctness and ease-of-use.

The major feature of the 0.3 release is per-message compression, which allows for text and binary to be sent in compressed form.

Here's a modified version of the Bitcoin price ticker example which enables compression. This spews every trade made on the GDax platform:

Using the per-message compression extension to the WebSocket spec with Lomond.

That's a lot of data coming through the WebSocket, which fortunately compresses very well.

Also in this release are a number of optimisations to reduce cpu and memory usage.

Lomond is fast approaching a 1.0 release, but is already very stable. It is part of the open source software that powers Dataplicity.

Apr 10, 2018
in Tech

Stream BTC prices over WebSockets with Python and Lomond

WebSockets is a great technology with many applications beyond the front end. I have a feeling that websocket APIs will become far more common in the web ecosphere.

I really like this example of how to interact with a WebSockets api with just-released Lomond 0.1.5.

Get the code from Lomond repos on Github.

This connects to the gdax websocket server and periodically prints the latest BTC price information. If you change the text "ticker" to "level2" it will spew out information on every trade. You could use this as the foundation for algorithmic trading or to keep an eye on your retirement fund (good luck).

Lomond is a websocket client library sponsored by Dataplicity. It has a focus on robustness and ease of use. continue reading…

Oct 9, 2017
in Tech

Lifting the Curtain on Asyncio Magic - Part 1

The new asyncio module introduced in Python3.4 is a nice addition to the Python standard library, especially when used with the async and await keywords introduced in Python3.5.

If you have read the official docs, you should hopefully be able to be able to write high performance network servers and clients with asyncio. But if something goes wrong, you may find it hard to debug. This is no reflection on asyncio; concurrency and IO are deceptively complex things, and asyncio is the point they meet.

I suspect that the problem with debugging asyncio projects is that async code doesn't run itself. You are dependant on a rather large body of code to run your code for you, and the distinction between your code library code is far more blurry than you may be used to. continue reading…

Aug 7, 2017
in Tech

Amazon S3 Filesystem for Python

I'd like to announce an new Python module to make working with Amazon S3 files a whole lot easier.

The S3FS class in fs-s3fs wraps an Amazon S3 bucket in a PyFilesystem interface. There was an S3FS class built in to the first version of PyFilesystem, but it had suffered from using an older version of 'boto' (Amazon's S3 interface) and was in need of maintenance. The new version is up-to-date with PyFilesystem2 and boto3, and works with Python2.7 and Python3.X.

If you aren't familiar with PyFilesystem, it is a common interface to anything that resembles a filesystem. Amazon S3 isn't quite a full filesystem, but close enough that it can be made to work in the same way as the files and directories on your local drive (or any other supported filesystem for that matter). continue reading…

Jun 11, 2017
in Tech

PyFilesystem 2.0.4 Released

I've released version 2.0.4 of PyFilesystem.

PyFilesystem is an abstraction layer for filesystems (or anything that resembles a filesystem). See this post for more details.

This version adds support for Python3.6 os.pathlike to the OSFS class. This is in oft-requested feature, but it only affects the OSFS constructor; since FS methods use a standardized path format.

Also in the version is a contribution by Martin Larralde which implements an elegant extension mechanism. Essentially, if you layout your filesystem implementation according to the convention, then PyFilesystem will be aware of it automatically. So open_fs('awesomefs://foo/bar') would work without explicitly adding awesomefs to the core library. continue reading…

Jun 1, 2017
in Tech

Announcing Lomond, a WebSocket Client Library for Python

I'd like to announce the first official release of Lomond, a new WebSocket client library for Python. The development was sponsored by Dataplicity.

Lomond is not the first websocket client for Python, so why go to the effort of building another one? For our purposes, we needed a stand alone client that didn't need a framework to run. So that excludes the websocket client support in Tornado, aiohttp etc. The two libraries that were suitable for our product, websocket-client and ws4py, both had show-stopper bugs with ssl support; websocket-client would sometimes refuse to processes packets until additional data was received, and ws4py could lose entire packets. I'm sure both libraries could be fixed, but neither project appears to be actively maintained. continue reading…

Popular Tags