New Encrypted Notes Web Application

March 4th, 2015

The last two weekends I scratched a web development itch. I've been using KeepNote to store notes for years now. It's a nice simple desktop app, which I use to store details such as past addresses, account numbers, phone numbers etc. And more sensitive information like PIN numbers and passwords.

I configured KeepNote to store notes in Dropbox so that I don't risk losing anything. This has worked quite well, but I've always been frustrated that (a) I can't access my notes on my mobiles devices, and (b) I'm relying on a third party to keep my secrets.

My answer to this is a web application that stores notes on a server, but does the encryption in the browser (i.e. with Javascript). That way, there is no need to trust the provider. I'm not the first person to think of this (try having an original idea these days), there are some pretty good implementation of this idea. But I wanted something that is self-hosting, i.e. I could install on my own server, and I had some ideas about how the user interface should work.

In particular, I wanted to implement Sublime Text's fuzzy search. Essentially this allows you to filter the list of notes with a few key presses. For instance, if I have a note entitled ‘Social Security Number’, I can find it by typing ‘SSN’ or ‘SocSecNum’.

The site is currently live. Feel free to create a new encrypted notebook, but be aware that it is just for testing. Please don't use this to store the PIN number for your safety deposit box, or missile launch codes just yet. I will likely wipe the DB at some point.

I've create an test notebook which you can play with here:

http://notes.moyaproject.com/~will

Passphrase is: “where there is a will”

To be honest, I'm not sure what would happen if more than one person is editing a notebook at a time – but feel free to try it out.

The code is available on GitHub. I'm no cryptography expert, so I would appreciate someone who is to review the code…

 

A simple method for rendering templates with Python

February 15th, 2015

I never intended to write a template system for Moya. Originally, I was going to offer a plugin system to use any template format you wish, with Jinja as the default. Jinja was certainly up to the task; it is blindingly fast, with a comfortable Django-like syntax. But it was never going to work exactly how I wanted it to, and since I don't have to be pragmatic on my hobby projects, I decided to re-invent the wheel. Because otherwise, how do we get better wheels?

The challenge of writing a template language, I discovered, was keeping the code manageable. If you want to make it both flexible and fast, it can quickly descend in to a mass of special cases and compromises. After a few aborted attempts, I worked out a system that was both flexible and reasonable fast. Not as fast as template systems that compile directly in to Python, but not half bad. Moya's template system is about 10-25% faster than Django templates with a similar feature set.

There are a two main steps in rendering a template. First the template needs to be tokenized, i.e. split up in a data structure of text / tags. This part is less interesting I think, because it can be done in advance and cached. The interesting part is the following step that turns that data structure in to HTML output.

This post will explain how Moya renders templates, by implementing a new template system that works the same way.

Let's render the following template:

<h1>Hobbit Index</h1>
<ul>
    {% for hobbit in hobbits %}
    <li{% if hobbit==active %} class="active"{% endif %}>
        {hobbit}
    </li>
    {% endfor %}
</ul>

This somewhat similar to a Django or Moya template. It generates HTML with unordered list of hobbits, one of which has the attribute class="active" on the <li>. You can see there is a loop and conditional in there.

The tokenizer scans the template and generates a hierarchical data structure of text, and tag tokens (markup between {% and %}). Tag tokens consist of a parameters extracted from the tag and children nodes (e.g the tokens between the {% for %} and {% endfor %}).

I'm going to omit the tokenize functionality as an exercise for the reader (sorry, I hate that too). We'll assume that we have implemented the tokenizer, and the end result is a data structure that looks like this:

[
    "<h1>Hobbit Index</h1>",
    "<ul>",
    ForNode(
        {"src": "hobbits", "dst": "hobbit"},
        [
            "<li",
            IfNode(
                {"test": "hobbit==active"},
                [
                    ' class="active"'
                ]
            ),
            ">",
            "{hobbit}",
            "</li>",
        ]
     ),
    "</ul>"
]

Essentially this is a list of strings or nodes, where a node can contain further nested strings and other nodes. A node is defined as a class instance that handles the functionality of a given tag, i.e. IfNode for the {% if %} tag and ForNode for the {% for %} tag.

Nodes have the following trivial base class, which stores the parameters and the list of children:

class Node(object):
    def __init__(self, params, children):
        self.params = params
        self.children = children

Nodes also have an additional method, render, which takes a mapping of the data we want to render (the conext). This method should be a generator, which may yield] one of two things; either strings containing output text or an iterator that yields further nodes. Let's look at the IfNode first:

class IfNode(Node):
    def render(self, context):
        test = eval(self.params['test'], globals(), context)
        if test:
            yield iter(self.children)

The first thing the render method does is to get the test parameter and evaluate it with the data in the context. If the result of that test is truthy, then the render method yields an iterator of it's children. Essentially all this node object does is render its children (i.e. the template code between {% if %} and {% endif %}) if the test passes.

The ForNode is similar, here's the implementation:

class ForNode(Node):

    def render(self, context):
        src = eval(self.params['src'], globals(), context)
        dst = self.params['dst']
        for obj in src:
            context[dst] = obj
            yield iter(self.children)

The ForNode render method iterates over each item in a sequence, and assigns the value to an intermediate variable. It also yields to its children each pass through the loop. So the code inside the {% for %} tag is rendered once per item in the sequence.

Because we are using generators to handle the state for control structures, we can keep the main render loop free from such logic. This makes the code that renders the template trivially easy to follow:

def render(template, **context):
    output = []
    stack = [iter(template)]

    while stack:
        node = stack.pop()
        if isinstance(node, basestring):
            output.append(node.format(**context))
        elif isinstance(node, Node):
            stack.append(node.render(context))
        else:
            new_node = next(node, None)
            if new_node is not None:
                stack.append(node)
                stack.append(new_node)
    return "".join(output)

The render loop manages a stack of iterators, initialized to the template data structure. Each pass through the loop it pops an item off the stack. If that item is a string, it performs a string format operation with the context data. If the item is a Node, it calls the render method and pushes the generator back on to the stack. When the stack item is an iterator (such as a generator created by Node.render) it gets one value from the iterator and pushes it back on to the stack, or discards it if is empty.

In essence, the inner loop is running the generators and collecting the output. A more naive approach might have the render methods also rendering their children and returning the result as a string. Using generators frees the nodes from having to build strings. Generators also makes error reporting much easier, because exceptions won't be obscured by deeply nested render methods. Consider a node throwing an exception inside a for loop; if ForNode.render was responsible for rendering its children, it would also have to trap and report such errors. The generator system makes error reporting simpler, and confines it to one place.

There is a very similar loop at the heart of Moya's template system. I suspect the main reason that Moya templates are moderately faster than Django's is due to this lean inner loop. See this GutHub gist for the code from this post. You may also find Moya's template implementation interesting.

 

Long Time No See

February 10th, 2015

The feed for this Blog has been broken for a while. I knew it was down; somebody was kind enough to alert me on twitter. I've been procrastinating fixing it because a) the Django code I wrote to power this blog is old and crusty, and b) I've been busy with another project. I only got around to fixing it now, because I have something to announce. Stay tuned.

Can't believe the last time I posted a blog was 2013. Where does time go.

 

Hiring a Python web application developer

November 26th, 2013

My client is looking to hire a new Python developer, initially for an 8 month contract. It's a home working position, we communicate mostly via Skype / email / gtalk etc. Although we do meet up in meatspace from time to time, so ideally a candidate would be in the London / Oxford area.

You will be working with your truly. The projects I've been working on are in the server side of web-enabled devices. The web interface is written in Django, so you'll need the usual battery of front-end technologies; HTML, CSS, Javascript etc. We have a Twistd server which communicates with devices in the field, that my client produces. In the middle we have dynamic user interface generation from XML.

So there is some genuinely interesting technology there, and more such projects planned. We need someone who is a good problem solver with a general interest in web technologies. There's also the occasionally need work with data at the bits and bytes level, so a working knowledge of C that would be a plus.

See the Careers page on wildfoundry.com for the full details.

 

Finding the first bit set with Python

November 25th, 2013

Here's a Python gotcha that I spent some time tracking down. I'm writing it up in the spirit of saving developers a debugging headache in the future.

I had an integer with a single bit set, and I wanted to find the index of that bit. For example, 4 in binary is 00000100. The 1 is at the third position from the right, which should give an index of 2 – since the first position is 0.

You can do this in two ways; either check each bit in turn until you find a 1, or you can use math as a shortcut. I chose the math solution:

>>> import math
>>> myint = 4
>>> int(math.log(myint, 2))
2

Simple right? Finally staying awake in high school maths paid off. So simple that it was the last bit of code I suspected to be broken (spoiler: it was).

This is Python 2.7 which still has two types of integer; type int and arbitrary long integer type long. I was testing with ints because that's what you get when you type 4. However the numbers I was getting out of the Django db where longs. Look what happens with the above code when you use 4L rather than 4:

>>> myint = 4L
>>> int(math.log(myint, 2))
1

And that was the result of my headache. Longs and ints are generally interchangeable. But not in this case. math.log gives a different result with long and ints. Which we can see here.

>>> math.log(4, 2)
2.0
>>> math.log(4L, 2)
1.9999999999999998

That tiniest of rounding errors for the long version would be insignificant for most applications, but not if you are converting to an integer and discarding the fractional part. The fix is simple. Round the return value of math.log to the nearest whole.

>>> int(round(math.log(4L, 2)))
2

If you are working with Python 3, this problem goes away. Another reason to migrate if you have the option!

 

Instant Pygame for Python Game Development How-to

June 22nd, 2013
Instant Pygame for Python Game Development How-to

Instant Pygame for Python Game Development How-to

Packt Publishing have released Instant Pygame for Python Game Development How-to, a guide to getting started with PyGame, written by Ivan Idris. This title will help you get over the initial hurdles in setting up a PyGame environment and developing your own games.

I was the technical reviewer for this book.

 

Where are those Print statements?

December 9th, 2012

When I'm debugging Python I tend to sprinkle my code liberally with print statements. Trouble is, I forget which files I added those print statements to and have to waste a few minutes tracking them down. Time which could be better spent doing pretty much anything else.

Not sure why I didn't think of this before, but I hacked together something that automatically prints the file and line of the print statement next to the output.

Here's the code:

import inspect
import sys
import os


class DebugPrint(object):
    def __init__(self, f):
        self.f = f

    def write(self, text):
        frame = inspect.currentframe()
        filename = frame.f_back.f_code.co_filename.rsplit(os.sep, 1)[-1]
        lineno = frame.f_back.f_lineno
        prefix = "[%s:%s] " % (filename, lineno)
        if text == os.linesep:
            self.f.write(text)
        else:
            self.f.write(prefix + text)

if not isinstance(sys.stdout, DebugPrint):
    sys.stdout = DebugPrint(sys.stdout)

To use it, save it as ‘debugprint.py’ then imported it somewhere. That's all you need to do. Works on Python 2.7 and probably other 2.X Pythons.

Here's what happens if you do it from the console:

>>> import debugprint
>>> print "Hello"
[<stdin>:1] Hello

For print statements in your app, you will get something like:

[foo.py:22] Hello
 

“ http://t.co/K2QmB061 one more free #python book ”

0
 

“ Been spending a lot of time writing #Python daemons lately. I rather like it. #daemonic ”

0
 

“ Has anyone done for the email module, what @kennethreitz has done for http? i.e. make the api only as complex than it needs to be. #Python ”

0
 
 
© 2008 Will McGugan.

A technoblog blog, design by Will McGugan