It's All Geek to Me

What to do with Locidesktop?

So what to do with locidesktop.com? It's a desktop-like website bookmarking tool – if you haven't seen it, take a quick look at this example desktop.

I built Loci Desktop a few months ago and promoted it on a few geek sites. It's been running ever since, with no maintenance from myself, happily serving up start pages to a small number of regular users. There was a buzz when I promoted it, people were largely impressed, some were indifferent, but few ended up using it regularly. So now I'm left with a quandary.

I could try and promote it. But to what end? It's not like I need a certain number of visitors to cover the hosting. I'm using the same VPS as I am for my blog, and I designed Locidesktop to be ultra-low bandwidth anyway – so it effectively costs me nothing to run.

One option would be to sell the entire site outright, as the domain and technology rights. But there is currently no way of monetizing it and I doubt anyone would be interested as a commercial venture.

I could try and license it as a b2b service. A few people have commented that it would be a useful intranet service. I'm not sure about this, but it sounds plausible.

Alternatively, I could give back to the Django community and release it as open source, which I imagine would be the popular thing to do. Money isn't my primary motivator (a close second perhaps), so I wouldn't be averse to doing this. Thing is though, it would require work on my part to document it and maintain it, and I have other open source projects I would prefer to concentrate on. But I can't deny that it would be cool to see Locidesktop sites popping up over the interwebs.

Finally, I could just leave it as is. I'm pleased with how it turned out, and I have a few loyal users. Maybe I should just be satisfied.

Any options I haven't considered? Leave a comment…

PyFilesystem 0.3 released

I am pleased to announce a new version of PyFilesystem (0.3), which is a Python module that provides a common interface to many kinds of filesystem. Basically it provides a way of working with files and directories that is exactly the same, regardless of how and where the file information is stored. Even if you don't plan on working with anything other than the files and directories on your hard-drive, PyFilesystem can simplify your code and reduce the potential of error.

PyFilesystem is a joint effort by myself and Ryan Kelly, who has created a number of new FS implementations such as Amazon S3 support and Secure FTP, and some pretty cool features such as FUSE support and Django storage integration.

As an example of how awesome this package is, take a look at the following 6 lines of code, which creates a ramdrive:

from fs.osfs import OSFS
from fs.memoryfs import MemoryFS
from fs.expose import fuse

home_fs = OSFS('~/')
home_fs.makedir('ramdrive', allow_recreate=True)
fuse.mount(MemoryFS(), home_fs.getsyspath('ramdrive'))

If you run this, a directory called ramdrive will appear in your home folder, the contents of which are stored purely in memory.

I prepared a screencast that gives a quick demonstration of some features – because if a picture is worth a thousand words, this video must be worth fifteen thousand words a second:

PyFilesystem screencast from Will McGugan on Vimeo.

See the project page on google code for more information, including API docs. There are also a couple of blog posts that will give a some more context.

This release has reached a good level of stability and maturity. I'd like to invite as many Pythonistas as possible to check out this module and possibly contribute to the project.

Review of Django 1.2 E-commerce

I've worked with Django for more than two years now. The majority of the sites I have worked on have been social-networking or content based, but I have yet to do any serious work on a site where the main purpose is to advertise and sell products. So I when a copy of ‘Django 1.2 e-commerce’ landed on my desk I was intrigued by what it might cover that I hadn't been exposed to with other fields of Django development.

Django 1.2 E-commerce

The book starts out with a brief run-down of Django. The first chapter is more of a explanation of the philosophy behind Django, and definitely not a tutorial. Which I think is fair enough; if you are building an e-commerce site, you are probably a professional Python developer and there are plenty of books to get you up to speed with Django. In the second chapter, the author runs through a simple web-shop application with an inventory and a ‘buy now’ button – which seemed more like a confidence building exercise than anything else, but it does do a good job of demonstrating how simple it can be to build this kind of application with Django.

The subsequent chapters go in to detail regarding managing users, shopping-carts and taking payments with Google Checkout and Amazon services. I've never used these payment services, so I found the information particularly interesting. I like the author's approach of making generic views in order to share functionality across payment services, but I would have liked some more detail in to the APIs involved.

Chapter 6 was a surprise, it covers a variety of modules that you can use to add powerful search capabilities to your application. Django's database querying will only get you so far with searching, if you need more sophisticated searching of the kind you would expect from Google then you will need to integrate one of a number of external modules and services, which chapter 6 covers pretty well. The following chapter covers exposing data via several APIs, and creating PDF reports with ReportLab, but not in any great detail.

I was impressed with chapter 8, which covers writing JavaScript to create rich AJAX interfaces – something which is pretty much expected in a modern web-site. I would have preferred a more detail here, but only because I have a particular interest in front-end technologies. The next chapter explains how to integrate a Django application with Amazon payment services and S3 storage to sell digital goods, and goes it to more detail than other chapters.

The final chapter covers a number of options you will have for deploying your application. If your application is moderately sophisticated and has many components, deployment can be a tricky affair. This chapter explains how Python technologies such as Fabric, Buildout and Virtualenv can ease deployment headaches. It also covers serving the site with Apache and ‘mod_wsgi’.

The code snippets in this book are pretty good at demonstrating the subjects covered, but I did notice some quite glaring syntax errors in several of the code examples. The errors weren't subtle either; they would result in the code not even running. I suspect that many of them were likely to be the result of a non-technical editor re-formating the code and not a mistake on the part of the author, but there were also a number of programming errors and bad practices which were a little disapointing to see in a book aimed at professionals. For instance, the author consistently used the ‘is’ operator in place of the equality operator (they are not interchangable even though they may appear to be).

Overall, my impression of this book was favourable. It's definitely not a tutorial book in that its not going to teach you any new skills – since it covers so many technologies and doesn't go in to great detail about any of them. What it will do is give you a grounding of the components in an e-commerce system. If you are looking to build some kind of web-shop in Django then I would recommend this book. It's less of an essential purchase if you aren't working with e-commerce, but since many of the topics discussed in Django 1.2 e-commerce are applicable to other web-sites you may still want to check this book out.

Announcing Sore Thumb, a thumbnail and image processing module for Django

I recently worked on the re-design of 2 Degrees, which required a lot of image processing on thumbnails. The thumbnails where to be in a variety of different sizes, all with rounded corners and keylines on a selection of virtually identical off-white backgrounds and gradients. And they all had to work on IE6 *spit* without the transparency hack.

A variety of thumbnails generated by Sore Thumb

A lesser engineer may have told the front-end developer where to stick his rounded corners, but I didn't want see a grown man cry, so I built Sore Thumb, an on-the-fly thumbnail and image processing system for Django.

Sore Thumb uses a declarative method of defining thumbnails, similar to Django's model and form definitions. Here's an example of how to declare a thumbnail processor that produces a 120x100 pixel thumbnail with 10 pixel rounded corners and a dark grey keyline:

from sorethumb.djangothumbnail import DjangoThumbnail
from sorethumb.filters.defaultfilters import ThumbnailFilter
from sorethumb.filters.drawfilters import RoundedCornerFilter

class RoundedCornersEdged(DjangoThumbnail):
    format = 'png'
    filters = [ThumbnailFilter(120, 100),
               RoundedCornerFilter(10, border='#333')]

Once this class has been imported, the thumbnail processor will be available in templates via the sorethumb filter which takes a Django FileField and returns the url to the thumbnail.

For example:

{% load sorethumb %}

<img src="{{ profile.photo|sorethumb:"rounded_corners_edged" }} />

You will also need to add sorethumb to your INSTALLED_APPS for this to work.

That's pretty much all there is to working with Sore Thumb, see these examples for inspiration and the documentation for the details.

You can install sorethumb with easy_install, PIP, or directly from source:

easy_install -U sorethumb

Please let me know what you think of Sore Thumb. I'd rather not spend too much time maintaining it, but since most of the code has been in production for a number of months already, there shouldn't be many changes required. If you have any feature requests or bug-reports, now would be a good time to raise them while I have the time to do the work!

Django E-Commerce

Pakt publishing have released a new Django book called Django 1.2 E-Commerce, written by Jess Legg. I'll have a copy soon and post a review here.

In the meantime they are offering a free chapter of the book; Chapter 2 Setting up Shop in 30 minutes.

Exciting Python Developer Job

Meebo, the instant messenger in your browser company, are seriously looking for Python developers right now to work in Mountain View, California. From what I can gather they are expanding, and are building a number of sites in the Django framework.

Damn foreign keys, stealing our jobs and women

Django has support for Generic Foreign Keys, which let you reference one model instance from another, without knowing up-front what that model type is. The classic use for something like this is for a commenting system; you need generic foreign keys – or something like them – because you wouldn't want a commenting system that only worked with a single model.

If you have ever used generic foreign keys in Django, you will know that it is not quite transparent to the developer; a little effort is required to manage the various content types. I'll present here an alternative method to achieve this late binding of foreign keys that doesn't require storing the type of the object (as generic foreign keys do) and is completely transparent to the developer. I'm sure I'm not the first to think of this method, but I haven't yet seen it used in other Django projects.

Rather than store the type of object in a separate field, we can create a new model for each foreign key type we want to reference. For example; lets say we have a Rating model, and we want to rate Articles and Images – we could do this by generating a ArticlesRating model and a ImagesRating model with appropriate foreign keys. The easiest way to do this is with a function that returns a parameterized class definition.

Here's a snippet of code from a project I'm working on, that does just that:

rating.py

from django.db.models import Model, ForeignKey, IntegerField, Count, Avg
from django.db import IntegrityError
from django.contrib.auth.models import User

def make_rating_model(rated_model, namespace):

    class Rating(Model):

        user = ForeignKey(User)
        rated_object = ForeignKey(rated_model)
        vote = IntegerField(default=0, blank=True, null=False)

        class Meta:
            abstract=True
            db_table = u'rating_%s_%s' % (namespace, unicode(rated_model).lower())
            unique_together = ('user', 'rated_object')

        def __unicode__(self):
            return u"%s's rating of %s" % (self.user.username, unicode(self.rated_object))

        # Rest of the methods snipped for brevity
        # Contact me if you would like the whole class

    return Rating

This isn't a model definition, rather it is a function that create a model definition. You can call it multiple times to return a Rating model for each object you want a rating for. The function, make_rating_model takes two parameters; the name of the model you want to rate, and a string that is used to generate the table name, to avoid naming conflicts.

To create a rating object you would import ratings in your models.py file and add the following:

class ArticleRating(ratings.make_rating_model('Article', 'mysite')):
    pass

class ImageRating(ratings.make_rating_model('Image', 'mysite')):
    pass

Now if you syncdb you will get two completely independent models with essentially the same interface – which means you can write code that works equally well with model instances of either type.

This method doesn't quite replace generic foreign keys; if you don't know until runtime what model to reference, or if you require the objects to be in a single table, then you will still need generic foreign keys, but in my experience this is rarely the case.

Django Desktop

I implemented a theme system for locidesktop.com and thought it only fitting that the first theme I made was one for Django.

I present you with the Django themed Loci Desktop!

Here it is, in embedded form – although you really need to click the above link to fully appreciate it.

Talented front-end developer job for Web 2.0 company in Oxford

The company I work for, 2Degrees, is looking for a front-end developer to join our team.

Are you a CSS monkey?

We need a CSS monkey with a good working knowledge of browser quirks and the ability to get even IE6 looking good (although you don't have to like it). It would help if you don't run away screaming from Javascript and can play well with the code monkeys.

More details are below. Email the address at the bottom of the job description, and mention this blog!

Evolution of an Auto-Complete

My latests hobby-project has been pushed live, in invite-only beta form. Previously known as Links Desktop, I have now dubbed it Loci Desktop, after the Loci Method.

Auto-complete in Loci Desktop

One feature of Loci Desktop is that it will auto-complete URLs when you add new icons to your ‘desktop’. Auto-complete is one of those features that users expect these days. They want the app to figure out what they want with as few key-presses as possible – and quite rightly so, typing is such a chore!

The auto-complete system for Loci Desktop, in its initial state, was straight-forward to implement. The javascript and front-end was the most time-consuming part of the job, but the back-end Python code was trivial.

Amoeba

Alas, it was too slow to be practical. The list of URLs that I was auto-completing from came from a list of the top one million sites from Alexa.com, stored in MySQL and queried with the Django ORM. The query searched the urls for a substring, and sorted by the Alexa rank so the most popular sites were listed first.

Although it worked perfectly, the auto-complete code at the back-end hammered the server and took to long to return its result. Reducing the number of URLS to 100,000 helped, but didn't make it as usable as auto-complete in a desktop app.

Opposable Thumbs

I'm no expert on what goes on under the hood in a database, but the conclusion I came to was that there was no way that the DB could produce an index for substring searches on-the-fly, and had to resort to comparing the substring with every entry in the database. With a million entries, that could never be fast.

Caching helped, but only for URLs that were previously searched for. But it occurred to me that if the results for all possible searches were cached then auto-complete would be blisteringly fast. I almost dismissed that idea as crazy talk, but mulled it over anyway.

It turned out to be practical. There are a lot of substrings for any given URL. For example, “facebook” contains 8 one-character substrings, 7 two-character substrings ('fa', ‘ac’, ‘ce’, ‘eb’, ‘bo’, ‘oo’, ‘ok’), and so on. So there are going to be a log of substrings for each url – but there will be a lot of substrings common to many urls, and I only need to store 10 ‘hits’ for each substring.

Generating this substring index took quite a bit of brute force processing, but once uploaded to the server it means that I could use a single, extremely efficient query to generate the auto-completed urls. The query time went down from more than a second, to 0.002 seconds! A very satisfying result, which meant that the auto-complete would update almost as fast as I could type, at about 150 milliseconds per request.

Making Tools

Another optimization was to offload a bit of work to the client by caching in Javascript. It was trivial to implement, but not a particularity big win as it only speeded up auto-completed URLs that had been searched for previously (such as when you delete characters).

Geek here, make fire!

Although these optimizations made the auto-complete nice and fast, the small delay in receiving the first list of URLs meant that it wasn't obvious there was auto-complete if you hadn't used it. It would be preferable if the auto-complete selection appeared after the first key-press. So I generated a mapping of every letter and digit on to the corresponding list of urls and used that to auto-complete the first character, rather than make a round-trip to the server.

Making the first character auto-complete virtually instantaneous really made it feel snappier from the start. So a big win, for minimal effort.

Conclusion

Databases are highly tuned pieces of software, but you can get big wins if you massage your data in to a more efficient format!