Making App Engine More Pythonic

Lately, I was re-evaluating Google App Engine – the cloud computing platform – to see how feasible it would be for one pet project I’ve had in mind. It was pleasantly surprising overall, as the platform improved quite a lot while I wasn’t looking, since about a year and a half ago. Mostly interested in the Python part, I noticed that version 2.7 is now standard, lots of libraries are available out of the box, and it’s possible to use to pretty much any web framework you’d like to, such as Flask or Django.

Still, there are some quirks. App Engine SDK, for example, is a self-contained bundle with a bunch of Python packages that make it possible to run the development server with your app on your local machine. You don’t really “install” it into your Python interpreter, though.
Same goes for any additional, third party libraries your app may need. They must all be deployed along with it, as there is no setup.py or requirements.txt to specify your dependencies in. If you’re used to how e.g. Heroku handles dependencies, GAE’s way will undoubtedly be quite a letdown.

Good news are: you can still make it work sanely. By that I mean using virtualenv for development rather than your global, system-level interpreter, and keeping the code of any third party libraries out of your project’s repository. You may not get quite the same experience of pip install and pip freeze > requirements.txt but well… it’s close enough :)

Dependencies as Git submodules

So you have an application that requires some external libraries. Few of them are provided by App Engine itself, and you will be able to access them after you specify your requirement in app.yaml. Many times, however, you will want to tap into broader open source ecosystem, just like you’d like with any other Python app.

There is a way, fortunately, to include external libraries to go with your application without them cluttering your repository. Since the de facto standard for publishing code on the ‘net is to push it to a public Git repository, we can use Git submodules to “symlink” to those repositories. Our own Git repo won’t store any of their actual content, but only a list of URLs; the .gitmodules file.

If you held your breath at the mere mention of Git submodules, don’t panic. They get a lot of flak, that’s true, and many of their claimed shortcomings are quite genuine. All of them apply to the scenario where a main repo uses submodules to reuse shared subproject that is modified in conjunction with the main one.
As you have probably noticed, this is totally different than the setting we’re discussing here. When including an external dependency, the fact that Git submodule points to specific commit in the other repo is a feature, not bug. It’s the exact same reason why we should always put version numbers in requirements.txt: upgrading a third party library must never be accidental, or you risk breaking your code through unexpected API or behavior changes.

So, how to do it – use Git submodules, that is? You substitute pip install with git submodule add:

This will establish reference between the repo under given URL and a directory path inside your project, fetching the repo’s content in the process. But as you will quickly notice in $ git status, that content won’t become part of the working directory.
After all this talk about being explicit with your libraries’ version, you probably also want to checkout a correct release:

Otherwise, you will work off whatever the current HEAD happened to be, exactly how pip install flask would give you whatever is the newest release in PyPI.

Working alone from a single machine, this would set you up for the time being. For starting somewhere else, though, you need equivalent of pip install -r requirements.txt, i.e. a way to fetch all your libraries at once. Here’s where git submodule update comes handy:

It will both setup your freshly cloned repo to use submodules specified in .gitmodules files, as well as pull the submodules’ content.

There’s much more to Git submodules, of course, so if you want to gain much more thorough insight into them than this short overview, I recommend having a look at the Git book. And as with most things, $ man git submodule is always helpful.

virtualenv for it all

With dependencies seemingly in place, you might be quite disappointed trying to, you know, use them:

The reason for that is simple, though: the libraries are physically there on your disk, but they are not in your virtualenv’s $PYTHONPATH, so Python has no idea where to import them from. There are ways to solve this problem that I could ramble for a while about, but I will just go ahead and demonstrate a ready-made shell script which handles it all :)
You might need to tweak it, e.g. if your GAE SDK installation path is different than /opt/google_appengine, but otherwise it should be pretty straightforward. One caveat, though: the script should be re-run after adding a brand new library, as described in previous section:

As an added bonus, you will get dev_appserver and appcfg binaries inside your virtualenv’s ./bin, so you may remove App Engine’s SDK directory from your regular $PATH.

Deployment shenanigans

Setup of a local development environment generally ends here – you should be now ready to run your app through dev_appserver. What’s still missing is making your bundled libraries work with remote Python on actual App Engine instance. Sadly, there is no virtualenv in the cloud.

Instead, we need to revert to the glorified sys.path hacks. Before importing anything, we extend the actual PYTHONPATH so that it covers our third party libraries. If their directory layout is just like shown in the first section (lib/ root with subdirs for different libraries), the following shim will suffice to correctly bootstrap the import mechanics:

Place this in the root of your project’s source tree (outside the main Python package) and point the app.yaml to it:

With this, you may now deploy your app and see whether it works correctly. If you encounter problems, I recommend taking a look at Flask on App Engine Project Template. Even if you intend to use different web framework, the example code should be largely applicable.

Promise Objects in JavaScript

JavaScript’s default mode of operation is to rely heavily on callbacks: functions invoked when a longer operation (such as network I/O) finishes and delivers result. This makes it asynchronous, which is a notably different programming style than using blocking operations contained within threads (or their equivalents, like goroutines in Go).

Callbacks have numerous problems, though, out of which the most severe one is probably the phenomenon of “marching to the right”:

When using the (still) common style of providing a callback as the last argument to a function initiating an asynchronous operation, you get this annoying result of ever-increasing indentation as you chain those operations together. It feels like the language itself is telling you that it was not designed for such a complex stuff… Coincidence? ;)

But it gets worse. Operations may fail somewhere along the way, which is something you’d probably like to know about. Depending on the conventions your project, framework or environment uses, this could mean additional boilerplate inside the callbacks to distinguish success from error cases. This is typical in Node.js, where first argument of callback represents the error, if any:

Alternatively, you may be asked to provide the error handler separately; an “errback”, as it’s sometimes called. Splitting the code into small parts is great and everything, but here it means you’ll have two functions as arguments:

Giving them names and extracting somewhere outside may help readability a little, but will also prevent you from taking advantage of one of the JavaScript’s biggest benefits: superior support for anonymous functions and closures.