Different Kind of Engineering

Have you ever heard the term “architecture” applied to the realm of software? Pretty sure you have. Our discipline, at least on the surface level, seems to borrow a lot of nomenclature from fields related to the construction of buildings. We talk about designs, frameworks, foundations, sometimes even scaffolds or facades.

And, of course, there is this whole convention of referring to programmers as “engineers”. Guess it simply sounds too good as a job title to let it go!

But frivolous as it may be, there is something deeply wrong about drawing analogies between erecting real structures and hacking virtual ones. It’s not even about attributing undue significance to the “mere” activity of sitting behind a screen and hammering at the keyboard; at this point, software development is probably at least as important as civil engineering – perhaps even more so.

The problem is that thinking of developing software as analogous to constructing buildings promotes flawed, unsustainable approaches. What is even more insidious is that they appear perfectly reasonable at first, mature even. It seems intuitively right, after all, to think of software project as something akin to construction project. In both cases you need to satisfy certain requirements, accommodate for interactions with external environment, and fit everything within time and/or budget constrains.

If it seems so right, then why it is so wrong? Well, because the similarity is only apparent, and thus largely superficial.

Design can only be arrived at by building

How common it is to begin construction of a building before its blueprint is finalized? Although few shovels might have tilled the ground here and there, construction engineers generally do not begin laying foundations until the structure’s design is ready. Before even the first bricks form a nascent wall, you can have pretty good idea of how the final thing will look like.

Can you say the same about software project? Certain methodologies (*cough* waterfall) claim to achieve almost the same degree of robustness and predictability, but most of us know how patently ridiculous they are. Not only is the final shape of system unknown when you start writing it, but it is practically unknowable. The design only reveals itself while you are already building, and this is something I’m sure would make the blood of traditional engineers run cold.

Laws of physics are of no concern

There are reasons for those unknowns, however. While real world engineering deals with real world limitations, bits and pixels are often completely unconstrained by those. More than by climate conditions, size of electromagnetic spectrum or the speed of light, software systems are limited by other, existing systems and processes: not necessarily software ones, but often unstable and changing.
As a result, dealing with them can get arbitrarily complex, not to mention that it transitively affects future systems interacting with us.

In comparison, the ground you build upon is quite literally rock solid.

Code is not concrete

Thankfully, there is one aspect of programming that makes all those obstacles somewhat surmountable. Unlike skeletons of skyscrapers, made of reinforced concrete, code is always modifiable. Regardless of how far you’ve progressed with current implementation, you can change and redesign it. Sure, there is typically a requirement to support outside clients through existing API but rarely, if ever, you are limited in how you’re gonna support them.

Frankly, this is something that I often see people struggling to come to terms with. Maybe because of those construction analogies I’ve mentioned, the structure of a program is often perceived as rigid and almost constant.
You are not allowed to tamper with it. You can only extend it in certain, predefined ways. And you better get it right the first time, as there is no going back. It’s almost as if the code was literally set in stone.

This, however, is a massive misconception. Code is meant to be poked and prodded; extended and removed; modified and reverted; and most importantly, hacked on and refactored. The notion of “technical debt” is a beautiful concept that captures all those mechanics. You don’t want to be in debt which is too large, because the interest rates (maintenance costs) will hinder you in the long run. But you also don’t want to have too small credit, or you won’t be able to make new investments (features) and expand your “business”.

Ask for Forgiveness, Not Permission

Even though it’s not a part of the Zen of Python, there is a widely accepted principle in Python community that reads:

It’s better to ask for forgiveness rather than permission.

For code, it generally means trying to do something and seeing whether it succeeded, rather than checking if it should succeed and then doing it. The first approach produces something like this:

and it’s preferred over the more cautious alternative:

What I realized recently is that there is much more general level of software development where the exact same principle can be applied. It’s not just about individual pieces of code, or catching exceptions versus checking preconditions. It extends to functions, modules, classes and packages – but more importantly, to the whole process of adding features, extending functionality or even conceiving totally new projects.

Taking a bit simplistic view, this process can be thought of having two extremes. On one end, there is a concept of pure “waterfall”. Every bit of work has to fit into predefined set of phases and no later phase can start before the previous one has completely finished. In this setting, design must be done upfront and all knowledge about the future product has to be gathered before coding starts.
By even this short description, it’s hard to stop thinking about all the hilarious ways the waterfall approach can end badly. Curiously, though, there are still companies that not only follow it but flaunt the fact publicly.

The other end is sometimes called “cowboy coding”, an almost derogatory term for churning out code without regard to pretty much any methodology whatsoever. Like in case of waterfall, it sometimes (very rarely) works, but it’s more so by accident rather than by design. And like pure waterfall, it also doesn’t really exist in nature, as long as more than one programmer is involved.

If we frame the spectrum like this, with the two completely opposite points at the edges, the natural tendency is to search for the middle ground. But if we subscribe to the titular principle of asking for forgiveness rather than permission, the right answer will clearly be shifted towards the “lean” side.
What is permission here? That’s a green light from design, architecture, requirements, UX and similar standpoints. You wouldn’t mind having all these before you start coding, but it is not absolutely crucial. Not having it, you can still do things: code, prototype, explore, and see how things pan out.
Even if you end up ripping it all apart, or maybe even fail to produce anything notable, you will still accrue useful knowledge. Say you have to throw away most of the week’s work because you’ve learned you need to do it differently. So what? With the insight you have gained along the way, it may very well take just a few hours now.

And so thou shalt be forgiven.