Practicality versus Purity for Python Templates

February 28th, 2008

There are a number of very powerful template languages available in Python. Some template languages, such as Genshi, allow complex Python statements and even full Python code within the template file. Others, such as Django templates, prefer to restrict templates to presentation only and do not allow general Python expressions within the body of the template.

In the context of a web framework, is it better to have the full expressiveness of Python, or restrict templates to presentation only?

 

Writing a Facebook Application with Python Pt. III

February 16th, 2008

This is the third part in a series of three posts about writing a Facebook application with Facebook. If you have not already ready done so, you should read the first and second parts first.

The technology behind Facebook applications is actually quite straight-forward. It is completely platform agnostic so you can build an app with any technology you might use to serve HTML content. My choice would be Python, but then I am a shameless Python fan-boy. Unfortunately there are still problems that any Facebook application developer will face, and most aren't technology related.

Facebook have solved a number of challenging problems and for the most part it is an elegant solution to adding third party applications to their own site. But Facebook isn't the grand free-for-all platform that the world wide web is -- Facebook must tread a fine line between allowing the application developer to generate revenue and avoid annoying the user so much that they abandon their accounts. Its understandable -- I wouldn't want an application popping up adverts or sending mails on my behalf -- but the rules they put in place tend to put the app developer at odds with Facebook admin.

For example, there is an API call that sends out notifications. When you receive a notification it displays a little text and link which you might use to inform a user that a friend has sent them something or done something else that is directly relevant to them. Facebook don't want applications flooding other users with these notifications, so it places restrictions on the number that can be sent out. The documentation says the application can 'send up to 40 notifications to the notifications page per user per day', which to to me is unclear whether an application can send 40 notifications per day on a user behalf or receive 40 notifications per day (apparently its the former). That used to be the case, but Facebook introduced a new system that restricts notifications even further. Applications receive a variable allowance of notifications that changes depending on how many users hide its notifications. The more users that hide your notifications, the less notifications an application may send out. The starting level seems to be 10 notification per day, a good bit under the original 40. This restriction strikes me as being excessively harsh on the application developer, and I'm not the only one who thinks so -- the forums are full of application developers venting their annoyance at this and other restrictions. In my opinion it would be far more sensible to restrict the number of notifications that can be received from an application, even one per user per day would be fine! I don't think anyone would be too annoyed by at most one notification per day send from a friend.

Facebook appear to be constantly tweaking and modifying their system to keep applications in check and make the content from applications as relevant as possible. And they prefer to do this by automated means, much to the chagrin of the application developers who feel they are being unfairly penalised by an flawed system. Unfortunately if you want a piece of Facebooks pie, you need to play by their rules. I suspect the gold-rush is over for Facebook applications, but as long as you keep on top the constantly shifting platform it could be a lucrative opportunity.

My own application doesn't seem to be doing too well. Perhaps because of the reasons I am posting here, or because it is a waste of time and people don't see the point. Whatever the reason, I am less inclined to create a Facebook app for becontrary.com, which was my original reason for looking in to it. It has started me thinking about how to make a better job of embedding an application inside a host web app, so writing a Facebook app may have been a good use of my time after all. Maybe I can do a better job, the poor schmucks at Facebook are lumbered with PHP. ;-)

 

Writing a Facebook Application with Python Pt. II

February 13th, 2008

For the first part series of post, click here for Part I.

Writing a Facebook application is in essence the same as writing any web application, only with an additional step where the output from your web app is processed and inserted in to a Facebook page. Although PHP seems the most popular choice (and is what Facebook itself is written in) you can use any of the Python frameworks to write an app. You could even roll your own if you were so inclined. I used Turbogears, but you could easily adapt this to another Framework.

Serving FBML

You can generate FBML with any Python templating system. I used Genshi which was quite happy to generate FBML for me as long as it was instructed to produce xml and not html, which you can do by adding format="xml" to the expose decorator. For example, here is the controller method for my stats page.

@expose(template="genshi:microbes.templates.stats", format="xml")
    def stats(self, *args, **kwargs):

        num_users = User.select().count()
        num_added_users = User.select(User.q.added==True).count()
        num_infections = Infection.select().count()

        return dict(num_users=num_users,
                    num_added_users=num_added_users,
                    num_infections=num_infections)

To see what that renders, take a look at the source of http://microbes.willmcgugan.com, which is naked FBML (not processed by Facebook).

Adding and Removing Users

The first page that the user requests probably wont be FBML at all. When the user adds your application, then Facebook will redirect to a URL defined in your app settings. If your app needs to store users then this is where you would add the users id to your database. You could then display a welcome page, or simply redirect to one of the pages in your application. Similarly, when a user removes your application the Facebook server requests another. This 'user removed' URL is never actually seen by the user so you don't need to return any meaningful content for it.

Using the Facebook API

For other pages within your application you need to do a little work to retrieve parameters from the page request that allow you to work with the API.

There are two Python modules to help with writing Facebook apps (that I know of). PyFacebook is a complete wrapper for the FB API and may be the obvious choice for writing a new application. However, the documentation was Django centric and I originally wanted to integrate an existing Turbogears application. So I decided to use minifb, which provides the necessary boiler-plate code for making calls to the Facebook API. If I write another application I would probably use PyFacebook to save a little time, but it turns out that doing it the hard way isn't that hard at all.

Before you use any parameters from a Facebook request, they need to be validated with a call to minifb.validate which takes your secret key (supplied by Facebook when you create an application) and a dictionary containing the request parameters. In Turbogears you would simply pass the keyword arguments from the controller method to minifb.validate. When a user has added your app, every call has at least two parameters supplied by the Facebook server; 'user' is the user id of the current user and 'session_key' is a string that identifies the current session which is needed for API calls. In the spirit of removing -- even minimal -- boilerplate I wrote a decorator to add to controller methods which does the validate step and optionally redirects to page if the user is not logged in.

_login_url = "http://apps.facebook.com/virtualmicrobes/"

def expose_fb(no_redirect=False):

    def decorate(f):

        def func(*args,  **kwargs):
            try:
                arguments = minifb.validate(FACEBOOK_SECRET_KEY, kwargs)
            except Exception, e:
                return str(e)

            if not no_redirect and "session_key" not in arguments:
                return '<fb :redirect url="%s" />'%_login_url

            ret = f(*args, **arguments)

            return ret

        func.__dict__.update(f.__dict__)
        return func

    return decorate

To use this decorator, just add it before Turbogears @expose decorator, which will make the controller appear just like any other request.

The minifb.call function can be used to make calls to the Facebook API. It takes the name of the API method you want to call, followed by your API key, secret key, session key and any additional parameters required by the method. It sends a POST request to the Facebook server and parses the returned JSON. The return value from minifb.call is a collection of basic Python data types, so it is very easy to work with. Generally API calls will return a list of integers / strings, or possibly a list of dictionaries.

Facebook Gotchas

The Facebook server imposes a timeout restriction on pages within your application. From my experiments this seems to be about 7 seconds, if the Facebook server doesn't receive a response from your web app within that time limit it will display 'page not found', or words to that affect. If your server is under strain and you are also making a lot of api calls then it is quite possible to go over the timeout. I initially had this problem, but it turned out to be due to a faulty DNS server which webfaction.com quickly fixed for me. So you may not experience this issue, but if you do I would suggest reducing the number of API calls you make if possible, and if you don't care about the return value of your API calls it would be worthwhile running them in a separate thread. I wrote the following helper function to do just that.

def fire_and_forget(callable, *args, **kwargs):

    def do_callable():
        try:
            callable(*args, **kwargs)
        except:
            pass

    thread = Thread(target=do_callable)
    thread.start()

If you wrap an API call with fire_and_forget it will run asynchronously and reduce the time to handle the request.

Cliffhanger

Hopefully I have convinced you that technology wise, writing a Facebook application is fairly straight-forward. Unfortunately there are other hardships which you will have to face when writing your app. I'll go over them in Part III.

 

Writing a Facebook Application with Python, Pt. I

February 9th, 2008

I promised to write up my experiences developing Virtual Microbes, a Facebook application. Its a lot to get through so I've decided to split it up in to installments. This installment is a lightning tour of the components in a Facebook application.

Facebook applications are basically content served up by any http server and seamlessly presented to users within Facebook pages. There are two ways this is done; either with an iframe tag or with Facebook Markup Language (FBML), which is basically HTML with additional tags to access Facebook features. Using an iframe may be the simplest option because you can just serve pages in the way you would with any web application, but FBML offers some additional capabilities that you wouldn't otherwise be able to take advantage of.

FBML

When the user requests a page within your application, the Facebook server maps the url on to a corresponding url on your server, retrieves the FBML, converts it to HTML, then inserts the content in to the Facebook page. Of course, its all transparent to the end user, who doesn't know and probably doesn't care where the content is coming from. FBML supports a large subset of HTML tags, and also CSS, so you can format and style pages as normal. The additional tags are used to insert content, such as user details / photos and various widgets within your content. For instance the <fb:name uid="12345"/> tag would insert the name of the Facebook user with the user id 12345.

FBJS

FBML supports FBJS, which is basically a slightly crippled form of Javascript which is re-written by the Facebook server to create a sandbox that prevents conflicts with Facebook's own JS code, HTML and CSS. It also cripples some Javascript features so that your code can only run in response to active events initiated by the user, such as clicks -- but not to events that the user hasn't directly caused. I guess Facebook do this to avoid annoying the user with impolite Javascript code, but it does mean you will probably have to modify your coding style to compensate. Simple code should work as you would expect it to, but you will probably find that your code breaks if you refer directly to nodes in the DOM, which contains IDs that are different from the originals in your FBML. For instance if you use getElementById to get an explicit element, it is unlikely to work.

Facebook API

In addition to FBML, Facebook also offers a web service that you can use to retrieve information regarding users and access various site features. For example, you can ask Facebook for a list of a users friends, send out notifications and emails, set FBML to be displayed in a users profile page and do various other site related tasks. You send the details of the API method you want to call and the server returns a response in XML or JSON.

FBQL

You can retrieve a lot of information with the Facebook API, but occasionally you may need to make more complex queries on the various pieces of information stored regarding users. Naturally Facebook doesn't want to give application developers the ability to drop or modify tables, so Facebook offers a cut down version of SQL that you can access via an API call. You can only SELECT with FBQL and there are restrictions regarding the columns you can use in the WHERE clause. For instance to query the user table, you must know either the users id or their name. I guess this is to prevent people from sucking down their entire database. But it does appear that if you know a users name you can retrieve a lot of information about them, that you wouldn't be able to access through the site itself. And you don't need to be an application developer either, you can user the test console to enter a query and retrieve the results. This feels like a privacy concern, but I guess applications couldn't add much value without this kind of information.

That covers the basics components involved in writing a Facebook application, in the next exciting installment of this post series, I'll cover how to write a Facebook application with a Python web framework.

 

Virtual Microbes

February 4th, 2008

I hacked together a Facebook application recently. I was looking in to integrating Becontrary.com with Facebook, but I had an idea for a simple app that would help me learn the ropes with FB. Its called Virtual Microbes -- the idea is that you give your friends a virtual disease and encourage them to pass it on. You gain a point each time you give a friend a microbe and each time a friend (or friend of friend) passes it on, which has the potential for exponential growth. Its completely pointless, and couldn't exist outside of the social networking sandbox, but it gave me the opportunity to experiment with integrating Turbogears with Facebook's application system. There was some pain initially, but ultimately it went smoothly. I'll write up my experiences with it, and give away some of the code soon.

 

Tag clouds look better sorted!

October 31st, 2007

I like the idea of tag-clouds and tags in general, but I had problems making them look good in becontrary.com. I experimented with changing the relative size and blending colors, as well as with the huge number of visual tweaks you can do with CSS, but the amorphous blob of words just didn't seem to fit with the nice neat columns I had. Until, that is, I sorted the tags by popularity, which made it a lot neater and enhanced the effect of blending the font size / color. I don't know why it didn't occur to me before. I don't see many sites doing this, most tag clouds are sorted alphabetically.

The image below shows a tag-cloud sorted by popularity (left) and just alphabetically (right). Which one do you think looks better?

Tags

(Click for a larger image.)

Update: A few people have commented that without alphabetical order it is difficult to pick out a tag you are looking for. Personally I have never used a tag cloud like that, its usually just a starting off point to explore the site. If I wanted something specific I would use the search function. Even if the tag-cloud is used to scan for specific tags, it becomes redundant if you have so many tags that you can only show a subset, which is the case for becontrary.com. So I chose form over function, consequences be damned!
 

Is there a CSS expert in the house?

October 28th, 2007

I've noticed that links in becontrary.com that have have a fragment (i.e. something after a #) don't always go to the exact location of the named anchor. I figured this was a Firefox bug originally, but I see the same thing in other browsers. I believe I have figured it out though. The browser changes the scroll position after the html is read, but before the stylesheets have been read. Once the browser has the CSS information the page updates, but because the CSS contains the dimensions of some elements, the named anchor changes position -- but the scroll position doesn't update accordingly. At least thats my working theory.  The only solution I can think of for this is to make all my style sheets inline -- but that would mean I wouldn't have the cache-related benefits of having them external. I can't be the first developer to be irritated by this. Can anyone offer a solution?

 

BeContrary out of Beta

October 20th, 2007
BeContrary.com seems to have been well-received. There hasn't been a huge amount of visitors, but I do have some loyal regulars that have made some great arguments. The site was on a few of the social bookmarking sites, which caused spikes in traffic. Annoyingly though, not all social bookmarking sites are created equal in terms of getting visitors. Visitors from Stumbleupon for example had a high bounce-rate because of its random site button. Other sites like metafilter.com and the Dilbert Blog brought more genuinely interested visitors.

The site was also the subject of a bit of vandalism. One user was repeatedly posting page after page of garbage text. Another user posted an argument where he professed his desire to perform fellatio (in slightly different terminology), which was kind of off-topic. So I hurriedly had to implement some anti-jerk technology to prevent flooding. I also had to add the ability to the admin page to completely wipe a users arguments / comments, something which I had naively thought I wouldn't need. The truth is though, that this type of site is vulnerable to vandalism, and if somebody wanted to make a nuisance of themselves then it wouldn't be difficult. All I could do is retroactively clean up the mess. I've had no problems with spam so far, because I implemented a number of anti-spam measures that using javascript to present different content to users than would be seen by a bot. It would be quite easy circumvent, but I can change the technique if spam becomes an issue -- or implement a captcha solution.

Turbogears and Webfaction hosting have consistently performed well, even when traffic spiked. I suspect that the number of visitors has never even come close to critical mass though, so perhaps my optimizations were a little premature. The content is mostly text-based and the largest page (the front page) takes up a meager 64.1K -- I've written emails bigger than that!

The XML or Text for Python Templates debates was quite popular for a while and has many compelling arguments for either site. I think I will try to include a number of Python / geek related debates. Some of the arguments in other debates people come up with are very funny. The following are two of my favorites:

" I would no more use a "Telepod of Doom" for transportation than I would a "Bus of Pain", a "Rickshaw of Destruction" or a "Personal Time Capsule of Discomfort". - In Telepods of Doom.

" Women are financially insolvent breeding mules. To expect that mewling milk spouts could afford the sort of meal required to flatter them is sheer fantasy. " In Going Dutch.

I have officialy declared the site non-beta, although I imagine I'll be tinkering with it for several weekends to come. Now I'm thinking about promotion. I really don't want to spend money on advertising, so I'd appreciate any suggestions of how to get more visitors!

 

XML or Text for Python Templates

October 15th, 2007

There are a number of (very good) templating systems and languages available for Python. They fall in to one of two camps; either they are XML based, like Genshi, or they are text based, like Mako. Most programmers favour one or the other, but there is far from a consensus over which is better.

I'd like to use this debate to gather reasons for using one over the other in the context of web development. I suspect there will be no clear winner, but it should serve as a useful resource for those faced with the decision!

http://www.becontrary.com/debates/xml_or_text_for_python_templates
 

Timed Caching Decorator

October 14th, 2007

Here's the caching decorator I mentioned in my previous blog entry about optimizing the front page of becontrary.com. It is pretty simple to use, you supply the same parameters to it as timedelata. So @timed_cache(minutes=30) would cache the result for half an hour. It is thread safe, so you can happily use it in the context of a web application, such as Turbogears. For many things it is a magical one-line speed-up, but there are some issues to be aware of. You can't use it to cache anything where the return value is context sensitive, such as SQLObject class instances -- because they are tied to a database context that wont exist at the second call. If you do want to cache such objects, then you should copy the information you need to a dictionary. Genshi templates make the difference completely transparent, so it is not much of a problem. Another issue is that parameters must be immutable, because they are used as keys in a dictionary.

from datetime import datetime, timedelta
from copy import deepcopy
from threading import RLock

def timed_cache(seconds=0, minutes=0, hours=0, days=0):

    time_delta = timedelta( seconds=seconds,
                            minutes=minutes,
                            hours=hours,
                            days=days )

    def decorate(f):

        f._lock = RLock()
        f._updates = {}
        f._results = {}

        def do_cache(*args, **kwargs):

            lock = f._lock
            lock.acquire()

            try:
                key = (args, tuple(sorted(kwargs.items(), key=lambda i:i[0])))

                updates = f._updates
                results = f._results

                t = datetime.now()
                updated = updates.get(key, t)

                if key not in results or t-updated > time_delta:
                    # Calculate
                    updates[key] = t
                    result = f(*args, **kwargs)
                    results[key] = deepcopy(result)
                    return result

                else:
                    # Cache
                    return deepcopy(results[key])

            finally:
                lock.release()

        return do_cache

    return decorate

if __name__ == "__main__":

    import time

    class T(object):

        @timed_cache(seconds=2)
        def expensive_func(self, c):
            time.sleep(.2)
            return c

    t = T ()

    for _ in xrange(30):
        time.sleep(.1)
        t1 = time.clock()
        print t.expensive_func('Calling expensive method')
        print "t - %i milliseconds"%int( (time.clock() - t1) * 1000. )

There are some other things to be aware of. Naturally it will use up more memory, because a copy of the result is stored for each combination of parameters. And the standard disclaimer applies, that you should check if there is actually a speed-up before using it in production code.

 
 
© 2008 Will McGugan.

A technoblog blog, design by Will McGugan