DreamPie with virtualenv

If you haven't heard about it, DreamPie is an awesome GUI application layered on top of standard Python shell. I use it for elaborate prototyping where its multi-line input box is a significant advance over raw, terminal UX of IPython.

However, up until recently I didn't know how to make DreamPie cooperate with virtualenv. Because it's a GUI program, I scoured its menu and all the preference windows, searching for any trace of option that would allow me to set the Python executable. Having failed, I was convinced that authors didn't think about including it - which was rather surprising, though.

But hey, DreamPie is open source! So I went to look around its code to see whether I can easily enhance it with an ability to specify Python binary. It wasn't too long before I stumbled into this vital fragment:

The conclusions we could draw from this anecdote are thereby as follows:

• It is indeed true that source code is often the best documentation...
• ...especially for open source programs where actual docs often suck.

With this newfound knowledge about dreampie arguments, it wasn't very hard to make it use current virtualenv:

And after doing some more research, I ended up adding the following line to my ~/.bash_aliases:

Now I can simply type dp to get a DreamPie instance operating within current virtualenv but independent from terminal session. Very useful!

Hacking Python Imports

...for fun and profit!

I'm still kind of amazed of how malleable the Python language is. It's no small feat to allow for messing with classes before they are created but it turns out to be pretty commonplace now. My latest frontier of pythonic hackery is import hooks and today I'd like to write something about them. I believe this will come handy for at least a few pythonistas because the topic seems to be rather scarcely covered on the 'net.

Importing: a simplistic view

As you can easily deduce, the name 'import hook' indicates something related to Python's mechanism of imports. More specifically, import hooks are about injecting our custom logic directly into Python's importing routines. Before delving into details, though, let's revise how the imports are being handled by default.

As far as we are concerned, the process seems to be pretty simple. When the Python interpreter encounters an import statement, it looks up the list of directories stored inside sys.path. This list is populated at startup and usually contains entries inserted by external libraries or the operating system, as well as some standard directories (e.g. dist-packages). These directories are searched in order and in greedy fashion: if one of them contains the desired package/module, it's picked immediately and the whole process stops right there.

Should we run out of places to look, an ImportError is raised. Because this is an exception we can catch, it's possible to try multiple imports before giving up:

While this is extremely ugly boilerplate, it serves to greatly increase portability of our application or package. Fortunately, there is only handful of worthwhile libraries that we may need to handle this way; json is the most prominent example.

More details: about __path__

What I presented above as Python's import flow is sufficient as description for most purposes but far from being complete. It omits few crucial places where we can tweak things to our needs.

First is the __path__ attribute which can be defined in package's __init__.py file. You can think of it as a local extension to sys.path list that only works for submodules of this particular package. In other words, it contains directories that should be searched when a package's submodule is being imported. By default it only has the __init__.py's directory but it can be extended to contain different paths as well.

A typical use case here is splitting single "logical" package between several "physical" packages, distributed separately - typically as different PyPI packets. For example, let's say we have foo package with foo.server and foo.client as subpackages. They are registered in PyPI as separate distributions (foo-server and foo-client, for instance) and user can have any or both of them installed at the same time. For this setup to work correctly, we need to modify foo.__path__ so that it may point to foo.server's directory and foo.client's directory, depending on whether they are present or not. While this task sounds exceedingly complex, it is actually very easy thanks to the standard pkgutil module. All we need to do is to put the following two lines into foo/__init__.py file:

There is much more to __path__ manipulation than this simple trick, of course. If you are interested, I recommend reading an issue of Python Module of the Week devoted solely to pkgutil.

Actual hooks: sys.meta_path and sys.path_hooks

Moving on, let's focus on parts of import process that let you do the truly amazing things. Here I'm talking stuff like pulling modules directly from Zip files or remote repositories, or just creating them dynamically based on, say, WSDL description of Web services, symbols exported by DLLs, REST APIs, command line tools and their arguments... pretty much anything you can think of (and your imagination is likely better than mine). I'm also referring to "aggressive" interoperability between independent modules: when one package can adjust or expand its functionality when it detects that another one has been imported. Finally, I'm also talking about security-enhanced Python sandboxes that intercept import requests and can deny access to certain modules or alter their functionality on the fly.

Bootstrap, a UI framework for the modern Web

It was almost exactly one year ago and I remember it quite vividly. I was attending an event organized by Google which was about the Chrome Web Store, as well as HTML5 and web applications in general. After the speaker finished pitching about awesomeness of this stuff (and how Chrome was the only browser that supported them all, of course), it was time for a round of questions and some discussion. I seized this opportunity and brought up an issue of user interface inconsistencies that plague the contemporary web apps. Because the Web as a platform doesn't really enforce any constraints on UI paradigms, we can experience all sorts of "creative" approaches to user interface. In their pursuit of novelty and eye candies, web designers often sacrifice usability by not adhering to well known interface metaphors, and shying away from uniform UI elements.

At that time I didn't really get a good answer that would address this issue. And it's an important one, given the rate at which web applications are springing to life and replacing their equivalent desktop programs. Does it mean we'll be stuck with this UI bonanza for the time being?...

Fortunately, there are some early first signs that it might not necessarily be so.

Enter Bootstrap

Few months later, in August 2011, Twitter released the first version of Bootstrap framework. Originally intended for internal use, this set of common HTML, CSS and JS patterns was made open source and released into the wild. The praise it subsequently gained is definitely well deserved, for it is a great set of tools for kickstarting development of any web-related project. Its features include:

• a flexible grid system for establishing a skeleton of the web page or app
• a set of great-looking styles for HTML form elements
• many complex UI components, like collapsible alerts, dropdowns, navigation bars, modal windows, and so on
• customizable set of CSS styles for typical markup elements, such as headers or tables

Along with universal acclaim came also the popularity: it is currently the most watched project on GitHub.

The value of consistency

However, some want to argue that being so popular has also an unanticipated, negative side. It makes the developers lazy, convinced they can get away without a "proper" design for their site or app. Even more: it allegedly shows disrespect for their users, as if the creator simply didn't care how does their product look like.

Does it compute? I don't think so. Do you complain if you find that any particular desktop application uses the very same looks for UI components, like buttons or list boxes?... Of course not. We learned to value the consistency and predictability that this entails, because it frees us from the burden of mentally adjusting to every single GUI app that we happen to use. Similarly, developers appreciate how this makes their work easier: they don't have to code dropdown menus or modal dialogs, which in turns allows them to spend more time on actual, useful functionality.

Sure, it didn't happen overnight when desktop OS' were emerging as software platforms. Also, there are still plenty of apps whose creators - willfully or unintentionally - chose not to adhere to the standards. Sometimes it's even for the good, as it allows for new, useful UI patterns to emerge and be adopted by the mainstream. The resulting process is still that of convergence, producing interfaces which are more consistent and easier to use.

Bootstrap shapes the Web

The analogous process may just be happening to the Internet, considered as a "platform" for web applications. By steadily raising in popularity, Bootstrap has a chance of becoming the UI framework for Web in general - an obvious first choice for any new project.

Of course, even if this happens, it would be terribly unlikely that it starts reigning supreme and making every website look almost exactly the same - i.e. transforming the Web into equivalent of desktop. What's much more likely is following the footsteps of mobile platforms. In there, every app strives to be original and appealing but only those that correctly balance usability with artsy design provide really compelling user experience.

It will not be without differences, though. Mobile platforms are generally ruled with iron (or clay) fist and have relevant UI guidelines spelled out explicitly. For Web it's very much not the case, so any potential "standardization" is necessarily a bottom-up process whose benefits have to be indisputable and clearly visible. Despite some opposition, Bootstrap seems to have enough momentum to really (ahem) bootstrap this process. It already wins hearts and minds of many web developers, so it may be just a matter of time.

If it happens, I believe the Web will be in better place.

Prison Escape: Game from IGK Compo

Yesterday I came back from IGK conference (Engineering of Computer Games) in Siedlce. Among few interesting lectures and presentations it also featured a traditional, 7-hour long game development contest: Compo. Those unfamiliar with the concept should know that it's a competition where every team (of max. 4 people) has to produce a playable game, according to particular topic, e.g. theme or genre. When the work is done, results are being presented to the public while a comittee of organizers votes for winners.

As usual, I took part in the competition along with Adam Sawicki "Reg" and Krzystof Kluczek "Krzysiek K.". Topic for this year revolved around the idea of "escape" or "survival" kind of game, so we designed an old school, pixel-artsy scroller where you play as a prisoner trying to escape detention by running and avoiding numerous obstacles. We coded intensely and passionately, and in the end it paid off really well because we actually managed to snatch the first place! Woohoo!

A whopping amount of 15 teams took part in this year's Compo, so it might take some time before all the submissions are available online. Meanwhile, I'm mirroring our game here. Please note that it uses DirectX 9 (incl. some shaders), so for best results you should have a non-ancient GPU and Windows OS. (It might work under Wine; we haven't tested it).

  [2012-04-01] Prison Escape (5.8 MiB, 125 ściągnięć)

Working within Temporary Directory

Few days ago I needed to write a script which was supposed to run inside a temporary directory. The exact matter was about deployment from an ad hoc Git repository, and it's something that I may describe in more detail later on. Today, however, I wanted to focus on its small part: a one that (I think) has neatly captured the notion of executing something within a non-persistent, working directory. Because it's a very general technique, I suppose quite a few readers may find it pretty useful.

Obtaining a temporary file or even directory shouldn't be a terribly complicated thing - and indeed, it's very easy in case of Python. We have a standard tempfile module here and it serves our needs pretty well in this regard. For one, it has the mkdtemp function which creates a temporary directory and returns path to it:

That's what it does. What it doesn't do is e.g ensuring a proper cleanup once the directory is not needed anymore. This is especially important on Windows where the equivalent of /tmp is not wiped out at boot time.
We also wanted our fresh temp directory to be set as the program's working one (PWD), and obviously this is also something we need to manually take care of. To combine those two needs, I think the best solution is to employ a context manager.

Context manager is basically a fancy name for an object that the with statement can be applied upon. You may recall that some time ago I wrote about interesting use cases for the with construct. This one could also qualify as such, but the principles are very typical. It's about introducing a scope where some resource (here: a temporary directory) remains accessible as long as we're inside it. Once we leave the with block, it is cleaned up - just like file handles, network sockets, concurrent locks and plenty of other similar objects.

But while semantics are pretty clear, there are of course several ways to do this syntactically. I took this opportunity to try out the supposedly simplest one which I learned recently on local Python community meet-up: the contextlib library. It includes the contextmanager decorator: a simple and clever way to write with-enabled objects as simple functions. It is based on particular usage of yield statement which makes it very interesting even by itself.

So without further ado, let's look at the final solution I wanted to present:

As we can see, yield divides this function into two parts: setup and cleanup. Setup will be executed when we enter the with block, while cleanup will run when we're about to exit it. By the way, this scheme of multiple entry and exit points in one function is typically referred to as coroutine, and it allows for several very intriguing techniques of smart computation.

Usage of temp_directory function is pretty obvious, I'd say. Here's a simplified excerpt of the Git-based deployment script that I used it in:

Note how the meaning of '.' (current directory) shifts depending on whether we're inside or outside the with block. Users of Fabric (Python- and SSH-based remote administration tool) will find this very similar to its cd context manager. The main difference is of course that directory we're cd-ing to is not a predetermined one, and that it will disappear once we're done with it.

How Does this Work (in JavaScript)

Many caveats clutter the JavaScript language. Some of them are quite hilarious and relatively harmless, but few can get really nasty and lead to insidious bugs. Today, I'm gonna talk about something from the second group: the semantics of this keyword in JavaScript.

Why's this?

It is worth noting why JS has the this keyword at all. Normally, we would expect it only in those languages which also have the corresponding class keyword. That's what C++, Java and C# have taught us: that this represents the current object of a class when used inside one of its methods. It only makes sense, then, to use this keyword in a class scope, denoted by the class keyword - both of which JavaScript doesn't seem to have. So, why's this even there?

The most likely reason is that JavaScript actually has something that resembles traditional classes - but it does so very poorly. And like pretty much everything in JS, it is written as a function:

Here, the Greeting is technically a function and is defined as one, but semantically it works more like constructor for the Greeting "class". As for this keyword, it refers to the object being created by such a constructor when invoked by new statement - another familiar construct, by the way. Additionally, this also appears inside greet method and does its expected job, allowing access to the text member of an object that the method was called upon.

So it would seem that everything with this keyword is actually fine and rather unsurprising. Have we maybe overlooked something here, looking only at half of the picture?...

Well yes, very much so. And not even a half but more like a quarter, with the remaining three parts being significantly less pretty - to say it mildly.

Adding Recursive Depth to Our Functions

I suppose it this not uncommon to encounter a general situation such as the following. Say you have some well-defined function that performs a transformation of one value into another. It's not particularly important how lengthy or complicated this function is, only that it takes one parameter and outputs a result. Here's a somewhat trivial but astonishingly useful example:

Depending on what happens in other parts of your program, you may find yourself applying such function to many different inputs. Then at some point, it is possible that you'll need to handle lists of those inputs in addition to supporting single values. Query string of URLs, for example, often require such treatment because they may contain more than one value for given key, and web frameworks tend to collate those values into lists of strings.

In those situations, you will typically want to deal just with the list case. This leads to writing a conditional in either the caller code:

or directly inside a particular function. I'm not a big fan of similar solutions because everyone do them differently, and writing the same piece several times is increasingly prone to errors. Quite not incidentally, a mistake is present in the very example above - it shouldn't be at all hard to spot it.

In any case, repeated application calls for extracting the pattern into something tangible and reusable. What I devised is therefore a general "recursivator", whose simplified version is given below:

As for usage, I think it's equally feasible for both on-a-spot calls:

as well as decorating functions to make them recursive permanently. For this, though, it would be wise to turn it into class-based decorator, applying the technique I've described previously. This way we could easily extend the solution and tie it to our needs.

But what are the specific ways of doing so? I could think of some, like:

• Recursing not only on lists, bit also on mappings (dictionaries) and applying the function to dictionary values. A common use case could be a kind of sanitization function for preparing values to be serialized, e.g. by turning datetimes into ISO-formatted strings.
• Excluding some data types from recursion, preventing, say, sets from being turned into lists, as obviously sets are also iterable. In more general version, one could supply a predicate function for deciding whether to recurse or not.
• Turning recursive into generator for more memory-efficient solution. If we're lucky to program in Python 3.x, it would be a good excuse to employ the new yield from construct from 3.3.

One way or another, capturing a particular concept of computation into actual API such as recursive looks like a good way for making the code more descriptive and robust. Certainly it adheres to one of the statements from Zen: that explicit is better than implicit.