Spatial Bookmarking Service goes Open Source

January 9th, 2011

Locidesktop was my coffee shop coding project of last year. I was quite pleased with the results. Locidesktop.com has been happily serving link desktops to some loyal visitors for months now – with no maintenance required on my part (a good thing because I've been busy with other projects).

Rather than let the project stagnate while I do other things, I have released it as open source with the hope that other Python/Django developers will pick it up and add new features. I had never really anticipated that anyone other than myself would see the code, so it is lacking in comments and documentations, but there are some nice features that may be of interest. For instance, there's a system to expose a REST API that factors out a lot of Python bioler-plate code and corresponding javascript code exposes the desktop functionality to the browser.

There's also a pretty powerful caching system that makes rendering a desktop blindingly fast. I may have gone overboard with that, given my traffic rates. But I guess it is best to have more capacity than you need.

If you want to look through the code or fork the project, it is available on Github:

Locidesktop code

The only thing missing from the open source version is a single-site theme I purchased from themeforest.net, used in a few pages such as the about & privacy urls. The desktops themselves are identical to the live site because I did those myself. You can tell because of the minimalist design reflecting my artistic abilities (minimal).

There's a README in the project that will help you get started, but other than that you may have to figure things out for yourself. It's a pretty standard Django project, although I do use Jinja templates rather than Django templates. If you have any questions, please ask them in the comments, so at least there is a central repository for issues.

For more information on Locidesktop, see my previous posts on the subject.

Update: I had to recreate the repos to remove a bunch of temp files and mercurial data. If you forked before 9am GMT, 10th Jan, you may want to get this new repos.

 

What to do with Locidesktop?

June 30th, 2010

So what to do with locidesktop.com? It's a desktop-like website bookmarking tool – if you haven't seen it, take a quick look at this example desktop.

I built Loci Desktop a few months ago and promoted it on a few geek sites. It's been running ever since, with no maintenance from myself, happily serving up start pages to a small number of regular users. There was a buzz when I promoted it, people were largely impressed, some were indifferent, but few ended up using it regularly. So now I'm left with a quandary.

I could try and promote it. But to what end? It's not like I need a certain number of visitors to cover the hosting. I'm using the same VPS as I am for my blog, and I designed Locidesktop to be ultra-low bandwidth anyway – so it effectively costs me nothing to run.

One option would be to sell the entire site outright, as the domain and technology rights. But there is currently no way of monetizing it and I doubt anyone would be interested as a commercial venture.

I could try and license it as a b2b service. A few people have commented that it would be a useful intranet service. I'm not sure about this, but it sounds plausible.

Alternatively, I could give back to the Django community and release it as open source, which I imagine would be the popular thing to do. Money isn't my primary motivator (a close second perhaps), so I wouldn't be averse to doing this. Thing is though, it would require work on my part to document it and maintain it, and I have other open source projects I would prefer to concentrate on. But I can't deny that it would be cool to see Locidesktop sites popping up over the interwebs.

Finally, I could just leave it as is. I'm pleased with how it turned out, and I have a few loyal users. Maybe I should just be satisfied.

Any options I haven't considered? Leave a comment…

 

Django Desktop

December 20th, 2009

I implemented a theme system for locidesktop.com and thought it only fitting that the first theme I made was one for Django.

I present you with the Django themed Loci Desktop!

Here it is, in embedded form – although you really need to click the above link to fully appreciate it.

 

When did I become the Javascript guy?

December 3rd, 2009

I have been busy working on locidesktop.com lately. One popular feature request from the beta testing was to use website favicons, so that links become more readily identifiable – which I have implemented after several late nights, and abusing my quad core.

3D Favicons example

A few of the 3D Favicons used in locidesktop.com

I built a pipeline that downloads favicons, extracts PNGs which are then processed in to a 3D scene and rendered for missing resolutions (.ICO files can contain multiple resolutions). I've tweaked the way that the 3D icons are produced, and they are significantly better than my first attempt – although some do come out better than others.

The 3D icons are integrated in to locidesktop so that they are pulled in whenever you add a site to your desktop. At the moment I have about 20,000 of them, and I'll be adding more in the future.

The desktop editing interface has been polished considerably, and I think that it is pretty slick now, although there are still plenty of improvements to be made, especially in the area of context sensitive help.

If you really want a beta invite, let me know.

I've also created a test account so that I can demo the features of locidesktop.com without a login. The url is http://locidesktop.com/test/ to view the test desktop, and http://locidesktop.com/test/default/edit/ to play with desktop editing (but you won't be able to save).

As always, I welcome feedback…

 

Turning website favicons in to 3D

October 9th, 2009

The feedback I recieved from Reddit about locidesktop.com (my hobby project) was encouraging.

If you would like to join the beta program for locidesktop, please leave a comment below…

One of the comments pointed out that although there is a large choice of icons available, there isn't always a clear recognizable image for each site, and it would be nice if locidesktop would use ‘favicons’. I had considered using favicons previously, but rejected the idea because they are just 16x16 pixels in size, and I wanted to use large images for icons.

I didn't want blurry scaled icons either, and I may have abandoned the idea if a Reddit user hadn't pointed me at this which recommended embracing the pixelated look of favicons for use at desktop icons. I figured I could take this idea a step further and render 3D images from any given 16x16 image, using a combination of Python, Mako templates and Povray – the same combination of technologies I used for my (now defunct) 3D pie chart project.

 

Evolution of an Auto-Complete

September 12th, 2009

My latests hobby-project has been pushed live, in invite-only beta form. Previously known as Links Desktop, I have now dubbed it Loci Desktop, after the Loci Method.

Screenshot of auto-completing urls in locidesktop.com

Auto-complete in Loci Desktop

One feature of Loci Desktop is that it will auto-complete URLs when you add new icons to your ‘desktop’. Auto-complete is one of those features that users expect these days. They want the app to figure out what they want with as few key-presses as possible – and quite rightly so, typing is such a chore!

The auto-complete system for Loci Desktop, in its initial state, was straight-forward to implement. The javascript and front-end was the most time-consuming part of the job, but the back-end Python code was trivial.

Amoeba

Alas, it was too slow to be practical. The list of URLs that I was auto-completing from came from a list of the top one million sites from Alexa.com, stored in MySQL and queried with the Django ORM. The query searched the urls for a substring, and sorted by the Alexa rank so the most popular sites were listed first.

Although it worked perfectly, the auto-complete code at the back-end hammered the server and took to long to return its result. Reducing the number of URLS to 100,000 helped, but didn't make it as usable as auto-complete in a desktop app.

Opposable Thumbs

There are still some beta invites for Loci Desktop available. Contact me if you want one.

I'm no expert on what goes on under the hood in a database, but the conclusion I came to was that there was no way that the DB could produce an index for substring searches on-the-fly, and had to resort to comparing the substring with every entry in the database. With a million entries, that could never be fast.

Caching helped, but only for URLs that were previously searched for. But it occurred to me that if the results for all possible searches were cached then auto-complete would be blisteringly fast. I almost dismissed that idea as crazy talk, but mulled it over anyway.

It turned out to be practical. There are a lot of substrings for any given URL. For example, “facebook” contains 8 one-character substrings, 7 two-character substrings ('fa', ‘ac’, ‘ce’, ‘eb’, ‘bo’, ‘oo’, ‘ok’), and so on. So there are going to be a log of substrings for each url – but there will be a lot of substrings common to many urls, and I only need to store 10 ‘hits’ for each substring.

Generating this substring index took quite a bit of brute force processing, but once uploaded to the server it means that I could use a single, extremely efficient query to generate the auto-completed urls. The query time went down from more than a second, to 0.002 seconds! A very satisfying result, which meant that the auto-complete would update almost as fast as I could type, at about 150 milliseconds per request.

Making Tools

Another optimization was to offload a bit of work to the client by caching in Javascript. It was trivial to implement, but not a particularity big win as it only speeded up auto-completed URLs that had been searched for previously (such as when you delete characters).

Geek here, make fire!

Although these optimizations made the auto-complete nice and fast, the small delay in receiving the first list of URLs meant that it wasn't obvious there was auto-complete if you hadn't used it. It would be preferable if the auto-complete selection appeared after the first key-press. So I generated a mapping of every letter and digit on to the corresponding list of urls and used that to auto-complete the first character, rather than make a round-trip to the server.

Making the first character auto-complete virtually instantaneous really made it feel snappier from the start. So a big win, for minimal effort.

Conclusion

Databases are highly tuned pieces of software, but you can get big wins if you massage your data in to a more efficient format!

 

Links Desktop Teaser Screencast

August 1st, 2009

In past blogs I've hinted on the fact that I'm working on a Javascript application in my spare time. It's not quite done yet, and wont be live for a few more weeks, but I do have something that I can show off.

This project is best explained in the form of a screencast. This is the first screencast I have ever made and I'm not sure if I have done it right, or if my humble server will cope with serving FLV files. If the server doesn't manage to keep up, I'll probably Youtube it.

 
 
© 2008 Will McGugan.

A technoblog blog, design by Will McGugan