Two Exceptions to Rule Them All

There’s no better way to start a new year than a hearty, poignant rant. To set the bar up high right off the bat, I’m not gonna be picking on some usual, easy target like JavaScript or PHP. To the contrary, I will lash out on everyone-and-their-mother’s favorite language; the one sitting comfortably in the middle between established, mature, boring technologies of the enterprise; and the cutting edge, misbegotten phantasms of the GitHub generation.

That’s, of course, Python. A language so great that nearly all its flaws people talk about are shared evenly by others of its kin: the highly dynamic, interpreted languages. One would be hard-pressed to find anything that’s not simply a rehashed argument about maintainability, runtime safety or efficiency – concerns that apply equally well to Ruby, Perl and the like. What about anything specifically “pythonic”?…

Well, let’s talk about Python’s exceptions, shall we?

Breath of “choice”

A long and detailed discussion is bound to follow now, isn’t it? There are so many exception classes in Python. Dozens, in fact, and the number has also grown significantly between 2.x and 3.x:

With this broad spectrum to choose from, it’s surely possible to pick a fine-grained exception class for even a highly unusual situation that may happen in your code. And when something goes awry in the standard library, a similarly detailed exception you’ve caught should tell you volumes about the problem by merely looking at the exception class.

Why then, in any typical code, in at least 90% of the cases, the exception you can throw or need to catch is either TypeError or ValueError?!

Not in my type

Seriously, it’s as if those two were the only useful “abstract” exception types. Because quite frankly, they are: the other 30-odd Exception subclasses are mostly floating point, I/O or system exceptions. They very nicely cover and distinguish all the various external failures that your code may need to deal with.

But what about the internal ones, of your code or the libraries it uses? Often, they are even more diverse, more important, and also more harmful. Yet you have a grand total of two exception classes to signal them!

Moreover, if the language and the standard modules are any hints, these two are pretty much the only classes you are meant to use. Just look how resourceful a simple TypeError appears to be:

• Trying to “call” something which is not callable? Raises TypeError.
• Called a function with too few arguments? It’s a TypeError.
• Too many arguments? That’s also a TypeError!
• Haven’t supplied a required argument? Tsk, tsk, TypeError.
• Tried to pass a keyword argument to a function that accepts none? Yep, TypeError.
• The other way around, with positional argument where keyword is expected? Why, have this TypeError!
• Placing keyword argument before positional? TypeEr… Well, that’s actually SyntaxError. Helpful, is it not?

We’ve only covered errors related to function calls, by the way. We could produce a similar list for object creation, and extend it with few other, miscellaneous entries.

The crucial question would be still left unanswered, though: whose type is wrong here, anyway?

I don’t always raise tailored exceptions, but when I do…

Okay, it’s true not every fundamental mishap in Python results in TypeError or ValueError. There is also AttributeError for accessing non-existent attributes, as well as IndexError and KeyError.

Speaking of the latter two… Have you noticed how both of those exception refer to essentially the same situation? You botched the argument to [] operator (brackets), and you botched it in effectively the same manner: by taking it from outside of the allowed domain.
But somehow, it matters very much what is expected as the argument’s type. Is it number? Then it’s IndexError. Something else, usually a string? Then KeyError it is.

It is not accidental, of course. Indexing an array and performing a dictionary lookup look syntactically the same in Python, and given the typical interpreter’s overhead their internal differences are rarely relevant anymore. These two operations used to be very different, though, and they still very much are in C, C++, Java, etc. Even though it doesn’t have to, Python preserves this bit of computing “history” by having two separate exception classes.

Fine.

Just why it suddenly tries to be meticulous here, while simultaneously throwing the all-encompassing TypeError left and right?!

As a side note, IndexError and KeyError have a common base called LookupError. Which makes them absolutely great, a shining example of how an exception hierarchy should be designed.
Too bad it’s also an exceedingly rare example.

Why it matters?

The design of Python’s standard exception, as it is shown, leaves much to be desired. You could ask why is it even important, though, and it’s perfectly legitimate question.

In production code of most applications, not much time is spent on error capture and recovery. When an exception happens, there is seldom anything to recover anyway. Most often, the error either causes the whole program to quit, or fails the current transaction (such as HTTP request).
Under these circumstances, it doesn’t matter a whole lot what kind of exception was it exactly. It’s of secondary importance at best, relevant mostly to error reporting: what severity of a log entry to use, or maybe what HTTP status code to return. No one will be splitting hairs if a sensible default is used – like priority ERROR or code 500, respectively.

But sometimes, this stuff matters. When you’re writing tests for your application, you want to verify that in well-defined circumstances, and for specific class of data, the code behaves precisely as intended. For successful runs, the behavior will include its immediate output and any state changes it effects. We verify those by assertions on function results, mocks, or both.

For failures, though, we need to find out what exactly was the failure mode that the fixture triggered. Capturing and inspecting exceptions is the easiest way to accomplish that, but it’s contingent on them having reliable, distinguishing features. When nigh every exception is TypeError from the standard library, or ValueError from own and third party code, it all becomes significantly trickier. At worst, it turns into a choice between underspecified and fragile tests, because the optimal solution would require your test suite to speak goddamn English:

Similarly, writing a library intended for external use requires paying attention to all parts of the interface we’re exposing – including exceptions. They should allow to quickly and easily tell all possible error cases apart. Of course, the end user might not care about some or even all of them, but it’s always their call to make. Good library will therefore accommodate for any such decisions.

Parting recommendations

As the state of affairs regarding exceptions in Python looks so shoddy, what we can do as developers to help with ameliorating it? There is no magic wand to fix the standard library, short of maybe writing PEPs. Migrating to Python 3 also improves things very, very little.

Not everything is beyond our reach, though. The code you have definite control over is obviously the one that you write. Make sure you introduce more diversity into exceptions you raise, instead of surrendering to the malpractice of throwing ValueError whenever anything at all wrong happens. Domain-specific exceptions are cheap to make, and they can still inherit from the omnipresent ValueError to maintain compatibility.

Be Sociable, Share!

• 
• 

Be Sociable, Share!

• 
• 

• Tweet
•