Essential Coding Activities (That I No Longer Do)

Writing code is not everything there is in programming. But writing code comprises of much more than just typing it in. There is compiling or otherwise building it; running the application to see whether it works how it breaks; and of course debugging to pinpoint the issue and fix it. These are inherent parts of development process and we shouldn’t expect to be skipping them anytime soon…

Well, except that right now, I virtually pass on all of them. Your mileage may vary, of course, but I wouldn’t be surprised if many more developers found themselves in this peculiar position. I actually think this might be a sign of times, and that we should expect more changes in developer’s workflow that head in this very direction.

So, how do you “develop without developing”? Let’s look at the before mentioned activities one by one.

Compiling

Getting rid of the build step is not really inconceivable. There are plenty of languages that do not require additional processing prior to running their code. They are called interpreted languages, and are steadily gaining grounds (and hype) in the programming world for quite some time now.
Python, JavaScript and Ruby are probably the most popular among them. Given that I’m currently doing most of my development work in the first two, it’s no wonder I don’t find myself compiling stuff all that often.

But even if we’re talking about traditional, de facto compiled languages (like Java or C++), there’s still something missing. It’s the fact that you don’t often have to explicitly order your IDE to compile & build your project, because it’s already doing it, all the time.
I feel there’s tremendous productivity gain by shortening the feedback loop and having your editor/IDE work with you as your write the code. When you can spot and correct simple mistakes as you go, you end up having more time and cognitive power for more interesting problems. This background assistance is something that I really like to have at all times, therefore I’ve set it up in my editor for Python as well.

Running

The kind of programs I’m writing most often now – server-side code for web applications and backends – does not require another, seemingly necessary step all that often: running the app. As it stands, their software scaffolding is clever enough to detect changes in runtime and automatically reload program’s code without explicit prompting.

Granted, this works largely because we’re talking about interpreted languages. For compiled ones, there are usually many more hurdles to overcome if we want to allow for hot-swapping code into and out of a running program. Still, there are languages that allow for just that, but they are usually chosen because of reliability requirements for some mission critical systems.

In my opinion, there are also significant programming benefits if you can pull it off on your development machine. They are again related to making the cycle of writing code and testing it shorter, therefore making the whole flow more interactive and “real-time”. As of recently, we can see some serious pushes into this very direction. Maybe we will see this approach hitting mainstream soon enough.

Debugging

“Oh, come on”, you might say, “how can you claim you’ve got rid of debugging? Is all your code always correct and magically bug-free?…”

I wish this was indeed true, but so far reality refuses to comply. What I’m referring to is proactive debugging: stepping though code to investigate the state of variables and objects. This is done to verify whether the actual control flow of particular piece of code is the one that we’ve really intended. If we find a divergence, it might indicate a possible cause for a bug we’re trying to find and fix.

Unfortunately, this debugging ordeal is both ineffective and time consuming. It’s still necessary for investigating errors in some remote, test-forsaken parts of the code which are not (easily) traceable with other methods and tools. For most, however, it’s an obsolete, almost antiquated way of doing things. That’s mainly because:

• Post-mortem investigation is typically more than enough. Between log messages and stacktraces (especially if they contain full frames with local variables), you’re often very capable of figuring out what’s wrong, and what part of the code you should be looking at. Once there, the fix should be evident… usually :)
• Many (most?) fixes are made because of failed run of an automated test suite. In this scenario, you have even more information about the bug (like the exacting assert that fails), while the relevant part of the code is even easier to localize. You might occasionally drop into debugger to examine local variables of the test run, but you never really step through whole algorithms.
• It’s increasingly easier to “try stuff” before putting it into project’s codebase, reducing the likelihood of some unknown factor making our program blow up. If we have a REPL (Read-Eval-Print Loop) to test small snippets of code and e.g. verify assumptions about API we’re going to use, we can eliminate whole classes of errors.

It’s not like you can throw away your Xdb completely. With generous logging, decent test coverage and a little cautiousness when adding new things, the usefulness of long debugging sessions is greatly diminished, though. It is no longer mandatory, or even typical part of development workflow.

Whatever else it may be, I won’t hesitate calling it a progress.

Thinning Your Fat Fingers

I often say I don’t believe programmers need to be great typists. No software project was ever late because its code couldn’t be typed fast enough. However, the fact that developer’s job consists mostly of thinking, intertwined with short outbursts of typing, means that it is beneficial to type fast, therefore getting back quickly to what’s really important.
Yet, typing code is significantly different game than writing prose in natural language (unless you are sprinkling your code with copious amount of comments and docstrings). I don’t suppose the skill of typing regular text fast (i.e. with all ten fingers) translates well into building screens of code listings. You need a different sort of exercise to be effective at that; usually, it just comes with a lot of coding practice.

But you may want to rush things a bit, and maybe have some fun in the process. I recently discovered a website called typing.io which aims to help you with improving your code-specific typing skills. When you sign up, you get presented with a choice of about dozen common languages and popular open source projects written in them. Your task is simple: you have to type their code in short, 15-line sprints, and your speed and accuracy will be measured and reported afterwards.

The choice of projects, and their fragments to type in, is generally pretty good. It definitely provides a very nice way to get the “feel” of any language you might want to learn in the future. You’ll get to see a lot of good, working, practical code written in it – not to mention you get to type it yourself :) Personally, I’ve found the C listings (of Redis data store) to be the most pleasant to both read and type, but it’s pretty likely you will have different preferences.

The application isn’t perfect, of course: it doesn’t really replicate the typical indentation dynamics of most code editors and IDEs. Instead, it opts for handling it implicitly, so the only whitespace you get to type is line and word break. You also don’t get to use your text navigation skills and clipboard-fu, which I’ve seen many coders leverage extensively when they are programming.
I think that’s fine, though, because the whole thing is specifically about typing. It’s great and pretty clear idea, and as such I strongly encourage you to try it out!

Don’t Write Classes

On this year’s PyCon US, there was a talk with rather (thought-)provoking title Stop Writing Classes. The speaker might not be the most charismatic one you’ve listened to, but his point is important, even if very simple. Whenever you have class with a constructor and just one other method, you could probably do better by turning it into a single function instead.

Examples given in the presentation were in Python, of course, but the whole advice is pretty generic. It can be applied with equal success even to languages that are object-oriented to the extreme (like Java): just replace ‘function’ with ‘static method’. However, if we are talking about Python, there are many more situations where we can replace classes with functions. Often this will result in simpler code with less nesting levels.

Let’s see a few examples.

Inheriting for custom __init__

Sometimes we want to construct many similar objects that differ only slightly in a way their constructors are invoked. A rather simple example would be a urllib2.Request with some custom HTTP headers included:

That works, but it’s unnecessarily complex without adding any notable benefits. It’s unlikely that we ever want to perform an isinstance check to distinguish between CustomRequest and the original Request, which is the main “perk” of using class-based approach.

Indeed, we could do just as well with a function:

Note how usage doesn’t even change, thanks to Python handling classes like any other callables. Also, notice the reduced amount of underscores ;)

Patching-in methods

Even if the method we want to override is not __init__, it might still make sense to not do it through inheritance. Python allows to add or replace methods of specific objects simply by assigning them to some attribute. This is commonly referred to as monkey patching and it enables to more or less transparently change behavior of most objects once they have been created:

You will likely say that this look more hackish than using inheritance and/or decorators, and you’ll be correct. In some cases, though, this might be a right thing. If the solution for the moment is indeed a bit hacky, “disguising” it into seemingly more mature and idiomatic form is unwarranted pretension. Sometimes a hack is fine as long as you are honest about it.

Plain old data objects

Coming to Python from a more strict language, like C++ or Java, you may be tempted to construct types such as this:

An idea is to encapsulate some common piece of data and pass it along in uniform way. In compiled, statically typed languages this is a good way to make the type checker work for us to eliminate certain kind of bugs and errors. If we declare a function to take ContentType, we can be sure we won’t get anything else. As a result, once we convert the initial string (like "application/json") into an object somewhere at the edge of the system, the rest of it can be simpler: it doesn’t have to bother with strings anymore.

But in dynamically typed, interpreted languages you can’t really extract such benefits because there is no compiler you can instruct to do your bookkeeping. Although you are perfectly allowed to write analogous classes:

there is no real benefit in doing so. Since you cannot be bulletproof-sure that a function will only receive objects of your type, a better solution (some would say “more pythonic”) is to keep the data in original form, or a simple form that is immediately usable. In this particular case a raw string will probably do best, although a tuple ("text", "html") – or better yet, namedtuple – may be more convenient in some applications.

So indeed…

…stop writing classes. Not literally all of them, of course, but always be on the lookout for alternatives. More often than not, they tend to make code (and life) simpler and easier.

Hello World Fallacy

These days you cannot make more than few steps on the Web before tripping over yet another wonderful framework, technology, library, platform… or even language. More often that not they are promising heaven and stars: ease of use, flexibility, scalability, performance, and so on. Most importantly, they almost always emphasize how easy it is to get started and have working, tangible results – sometimes even whole apps – in very short time.

In many cases, they are absolutely right. With just the right tools, you can make some nice stuff pretty quickly. True, we’re still far from a scenario where you simply choose features you’d like to have, with them blending together automatically – even if some folks make serious leaps in that direction.
But if you think about it for a moment, it’s not something that we actually want, for reasons that are pretty obvious. The less effort is needed to create something, the less value it presents, all other things being equal. We definitely don’t expect to see software development reduced into rough equivalent of clicking through Windows wizards, because everything produced like that would be just hopelessly generic.

But think how easy it would be to get started with that…

And thus we come to the titular issue which I took liberty in calling a “Hello World” Fallacy. It occurs when a well-meaning programmer tries out a new piece of technology and finds how easy it is to do simple stuff in it. Everything seems to fall into place: tutorials are clear, to the point and easy to follow; results appear quickly and are rather impressive; difficulties or setbacks are few and far between. Everything just goes extremely well.. What is the problem, then?

The problem lies in a sort of “halo effect” those early successes are likely to create. While surveying a new technology, it’s extremely tempting to look at the early victories as useful heuristic for evaluating the solution as a whole. We may think the way particular tech makes it easy to produce relatively simple apps is a good indicator of how it would work for bigger, more complicated projects. It’s about assuming a specific type of scalability: not necessarily tied to performance of handling heavy load of thousands of users, but to size and complexity of the system handling it.

Point is, your new technology may not really scale all that well. What makes it easy to pick up, among other things, is how good it fits to the simple use cases you will typically exercise when you are just starting out. But this early adequacy is not an evidence for ability to scale into bigger, more serious applications. If anything, it might constitute a feasible argument for the contrary. Newbie-friendliness often goes against long-term usability for more advanced users; compare, for example, the “intuitive” Ribbon UI introduced in relatively recent version Microsoft Office to its previous, much more powerful and convenient interface. While I don’t stipulate it’s a pure zero-sum game, I think catering to beginners and experts alike is surely more difficult than addressing the needs of only one target audience. The former is definitely a road less traveled.

When talking about software libraries or frameworks, the ‘expert’ would typically refer to developer using the tech for large and long-term project. They are likely to explore most of the crooks and crannies, often hitting brick walls that at first may even appear impassable. For them, the most important quality for a software library is its “workaroundability”: how well it performs at not getting in the way between programmer and job done, and how hackable it is – i.e. susceptible to stretching its limits beyond what authors originally intended.

This quality is hardly evident when you’ve only done few casual experiments with your shiny new package. General experience can help a great deal with arriving at unbiased conclusion, and so can the explicit knowledge about the whole issue. While it’s beyond my limited powers to help you significantly to the former, I can at least gently point to the latter.

Happy hacking!

You Are Smarter than Quantum Physicists

Fairly recently, I started reading up on quantum mechanics (QM) to brush up my understanding of the topic and, quite surprisingly, I’ve found it ripe with analogies to my typical interests: software development. The one that stands out particularly well relates to the very basics of QM and the way they were widely misunderstood for many decades. What’s really amusing here is that while majority of physicists seem to have been easily fooled by how the world operates on quantum level, any contemporary half-decent software engineer, faced with problems of very similar nature, typically doesn’t exhibit folly of this magnitude.

We are not uncovering the Grand Scheme of Things every day, of course; what I’m saying is that we seem to be much less likely to come up with certain extremely bad answers to all the why? questions we encounter constantly in our work. Even the really hard ones (“Why-oh-why it doesn’t work?!”) are rarely different in this regard.

Thus I dare to say that we would not be so easily tricked by some “bizarre” phenomena that have fooled many of the early QM researchers. In fact, they turn out to be perfectly reasonable (and rather simple) if we look at them with programmer’s mindset. The hard part, of course, is to discover that such a perspective applies here, instead of quickly jumping to “intuitive” but wrong conclusions.

To see how tempting that jump can be, we should now look at one simple experiment with light and mirrors, and try to decipher its puzzling results.

A story of shy photons

The setup is not very complicated. We have one light source, two detectors and two pairs of mirrors. One pair consists of standard, fully reflective mirrors. Second pair has half-silvered ones; they reflect only half of the light, letting the other half through without changing its direction.
We arrange this equipment as shown in the following picture. Here, the yellow lines depict path the light is taking after being emitted from the source, somewhere beyond the left edge.

Source of this and subsequent images

But in this experiment, we are not letting out a continuous ray of light. Instead, we send out individual photons. We know (from some previous observations) that half-silvered mirrors are still behaving correctly in this scenario: they just reflect a photon about 50% of the time. Normal mirrors, obviously, are always reflecting all the photons.

Knowing this, we would expect both detectors to go off with roughly similar frequency. What we find out in practice is that only detector 2 is ever registering any photons, and no particle whatsoever reaches detector 1, at any time. (This is illustrated by a dashed line).

At this point we might want to perform a sanity check, to see whether we are really dealing with individual particles (rather than waves that can interfere and thus cancel themselves out). So, we block out one of the paths:

and now both detectors are going off, but not simultaneously. This indicates that our photons are indeed localized particles, as they appear to be only in one place at a time. Yet, for some weird inexplicable reason, they don’t show at detector 1 if we remove the barrier.

There are all sorts of peculiar conclusions we could come up with already, including the mere possibility of photon going both ways to have an effect on results we observe. Let’s try not to be crazy just yet, though. Surely we can establish which one of the two paths is actually being taken; it’s just a matter of putting an additional sensor:

So we do just that, and we turn on the machinery again. What we find out, however, is far from definite answer. Actually, it’s totally opposite: both detectors are going off now, just like in the previous setup – but we haven’t blocked anything this time! We just wanted to take a sneak peak and learn about the actual paths that our photons are taking.

But as it turns out, we are now preventing the phenomenon from occurring at all… What the hell?!

Cookies!

The term ‘cookie’ shall be familiar not only to programmers, but also to many of the more conscious, ordinary computer users. I’m not, unfortunately, talking about sweet pastry, but HTTP cookies.

Those cookies that we all know and love (if we’re web developers) or hate (if we’re overly paranoid about privacy) are not the only thing in computing to be known under this name, however. I learned this quite recently when talking to a friend of mine who is working in the realm of IT security. As it turns out, ‘cookie’ can refer to quite diverse array of different solutions, all unified through similar underlying concept.

Python Optimization 101

It’s pretty much assumed that if you’re writing Python, you are not really concerned with the performance and speed of your code, provided it gets the job done in sufficiently timely manner. The benefits of using such a high level language usually outweigh the cons, so we’re at ease with sacrificing some of the speed in exchange for other qualities. The feasibility of this trade is always relative, though, and depends entirely on the tasks at hand. Sometimes the ‘sufficiently fast’ bar might be hung quite high up.

But while some attitudes are clearly beyond Python’s reach – like real-time software on embedded systems – it doesn’t mean it’s impossible to write efficient code. More importantly, it’s almost always possible to write more efficient code than we currently have; the nimble domain of optimization has its subdivision dedicated specifically to Python. And quite surprisingly, performance-tuning at this high level of abstraction often proves to be even more challenging than squeezing nanoseconds out of bare metal.

So today, we’re going to look at some basic principles of optimization and good practices targeted at writing efficient Python code.

Tools of the trade

Before jumping into specific advice, it’s essential to briefly mention few standard modules that are indispensable when doing any kind of optimization work.

The first one is timeit, a simple utility for measuring execution time of snippets of Python code. Using timeit is often one of the easiest way to confirm (or refute) our suspicions about insufficient performance of particular piece of code. timeit helps us in straightforward way: by executing the statement in questions many times and showing average, as well as cumulative, time it has taken.

As for more detailed analysis, the profile and cProfile modules can be used to gain insight on CPU time consumed by different parts of our code. Profiling a statement will yield us some vital data about number of times that any particular function was called, how much time a single call takes on average and how big is the function’s impact on overall execution time. These are the essential information for identifying bottlenecks and therefore ensuring that our optimizations are correctly targeted.

Then there is the dis module: Python’s disassembler. This nifty tool allows us to inspect instructions that the interpreter is executing, with handy names translated from actual binary bytecode. Compared to actual assemblers or even code executed on Java VM, Python’s bytecode is pretty trivial to analyze:

Getting familiar with it proves to be very useful, though, as eliminating slow instructions in favor of more efficient ones is a fundamental (and effective) optimization technique.