will

Where there is a Will

Will McGugan's Blog

I am a freelance software engineer living in Glasgow Leicester.

I post mostly about tech (particularly Python) and photography.

Background: A code monkey

I've released version 2.0.4 of PyFilesystem.

PyFilesystem is an abstraction layer for filesystems (or anything that resembles a filesystem). See this post for more details.

This version adds support for Python3.6 os.pathlike to the OSFS class. This is in oft-requested feature, but it only affects the OSFS constructor; since FS methods use a standardized path format.

Also in the version is a contribution by Martin Larralde which implements an elegant extension mechanism. Essentially, if you layout your filesystem implementation according to the convention, then PyFilesystem will be aware of it automatically. So open_fs('awesomefs://foo/bar') would work without explicitly adding awesomefs to the core library. continue reading…

I'd like to announce the first official release of Lomond, a new WebSocket client library for Python. The development was sponsored by Dataplicity.

Lomond is not the first websocket client for Python, so why go to the effort of building another one? For our purposes, we needed a stand alone client that didn't need a framework to run. So that excludes the websocket client support in Tornado, aiohttp etc. The two libraries that were suitable for our product, websocket-client and ws4py, both had show-stopper bugs with ssl support; websocket-client would sometimes refuse to processes packets until additional data was received, and ws4py could lose entire packets. I'm sure both libraries could be fixed, but neither project appears to be actively maintained. continue reading…

Wildlife photography can require great hardship on the part of the photographer; days of trekking, battling the elements, biting insects, etc. — all while carrying 20 kilos of camera gear.

This was not the case for my recent trip to Costa Rica, which was as close to luxury you can get in the Rainforest, if you don't mind the lack of air-conditioning and occasional oversize bug in your cabin.

My girlfriend and I stayed in El Romanso, an eco-lodge on the Osa Peninsula. Located in primary rainforest overlooking the Pacific ocean, it is an beautiful place, abundant with wildlife. Most of which I photographed from the lodge restaurant, which had been constructed on stilts to put you at eye level with monkeys and toucans. continue reading…

I've just released version 2.0.3 of PyFilesystem.

PyFilesystem is an abstraction layer for filesystems (or anything that resembles a filesystem). See this post for more details.

New in this version is a TarFS filesystem contributed by Martin Larralde, which compliments ZipFS nicely.

Contributed by gpcimino, the copy module was extended new functionality to selectively copy only new files from one filesystem to another. Very useful for backups.

This is in addition to bugfixes, and improved documentation.

I recently I had the opportunity to speed up some Websocket code that was a major bottleneck. The final solution was 60X (!) faster than the first pass, and an interesting exercise in optimizing inner loops.

When you send data via a websocket it is masked with a randomly generated four byte key. This masking provides protection from certain exploits against proxies (details here). The algorithm is super simple; essentially each byte in the payload is XORed with a corresponding byte from the key.

The first version of my XOR mask function looked this:

That's the kind of unfussy and elegant code you want to write for a job interview. If that's the solution that first came to mind, then you know your Python. Alas, this code is as slow as it is beautiful. continue reading…

I like to imagine components of a Python web framework as personalities. Here's how I would imagine a conversation between a typical view and template would go down:

View Here's that data I just looked up.

Template Don't like it.

View What's wrong with it?

Template Format's wrong.

View You can change the format if you like.

Template Don't wanna. You change it.

View OK. Fine. I'll change it for you.

Which is why I wanted Moya templates to be very capable in processing data, but still keep them designer friendly. If the template data is sane then the template should be able to deal with it, without going back to the back-end developer. continue reading…

I'd like to announce version 2.0.0 of PyFilesystem, which is now available on PyPi.

PyFilesystem is a Python module I started some time in 2008, and since then it has been very much a part of my personal standard library. I've used it in personal and professional projects, as have many other developers and organisations.

If you aren't familiar with PyFilesystem; it's an abstraction layer for filesystems. Essentially anything with files and directories (hard-drive, zip file, ftp server, network filesystems etc.) may be wrapped with a common interface. With it, you can write code that is agnostic as to where the files are physically located.

Here's a quick example that recursively counts the lines of code in a directory: continue reading…

It's not often I come across something in Python that surprises me. Especially in something as mundane as string operations, but I guess Python still has a trick or two up its sleeve.

Have a look at this string:

How many possible sub-strings are in s? To put it another away, how many values of x are there where the expression x in s is true?

Turns out it is 2.

2?

Yes, 2.

The empty string is in the string "A". In fact, it's in all the strings.

Turns out the empty string has been hiding every where in my code.

Not a complaint, I'm sure the rationale is completely sound. And it turned out to be quite useful. I had a couple of lines of code that looked something like this: continue reading…

This year, Ellen and I took a trip, roughly following the North Coast 500 route.

The West coast of Scotland is stunning. As a Glasgow cabbie observed, if it wasn't for the weather and the midges of the Scottish West Coast, Brits wouldn't bother leaving the UK for their holiday. I can't comment on the weather (which was beautiful), but he may have a point about the midges.

Here are some photos from that trip.

Thistles at sunrise. Taken at around 5.30am. Worth getting up for?

This image is part of a much larger panorama that I would like to print for my wall in three parts.

The iconic Highland Cow, or Highland Coo as we call them in Scotland.

If you aren't familiar with midges, they are tiny biting flies. Their vileness is brilliantly illustrated with the following video...

I've always been intrigued by HDR, but some of my earlier attempts have been pretty horrific (like someone ate a pack of crayons and vomited on the screen).

Sunsets are one of the few subjects where HDR really works. Without merging a range of exposures you can never expose for both the sun and the foreground. So I was quite pleased to get the following shot (in Flores, Indonesia).

The trick in processing an HDR photo is to make it look like it is not HDR. At least not obviously HDR...