The overall direction where Python 3 is going might be a bit worrying, but it’s undeniable that the 3.0 line has some really nice features and quality-of-life improvements. What’s not to love about Unicode string literals enabled by default? Or the
filter all being generators? Neato!
There are also few lesser known points. One is the new
nonlocal keyword. It shares the syntax with the
global keyword, which would make it instantaneously fishy just by this connotation. However,
nonlocal looks genuinely useful: it allows to modify variables captured inside function’s closure:
What about Python 2.x, though – are we out of luck? Well, not completely. There are a few ways to emulate
nonlocal through other pythonic capabilities, sometimes even to better effect than the
nonlocal keyword would yield.
Speaking of yielding… As you have probably noticed right away, the example above is quite overblown and just plain silly. You don’t need to play functional just to count some values – you would use a loop instead:
Really, the previous version is just a mindless application of the classic Visitor pattern, which is another reason why you shouldn’t do that: pattern overuse is bad. This saying, Visitor obviously has its place: it’s irreplaceable when traversing more complicated structures in more bureaucratic languages. A simple list of numbers in Python is the direct opposite of both of these characteristics.
Complex data structures exist in any language, however. How would we run some Python code for every node in a tree, or maybe graph? Unrolling the DFS or BFS or whatever traversal algorithm we use certainly doesn’t sound like an elegant and reusable approach.
But even then, there is still no need for functions and closures. We can easily get away with the simple
for loop, if we just find a suitable iterable to loop over:
bst_nodes function above is not black magic by any stretch. It’s just a simple example of generator function, taking advantage of the powerful
This works because both
bst_nodes functions are executed “simultaneously”. That has the same practical effect as calling the visitor function to process a node, only the “function” is concealed as the body of
Language geeks (and Lisp fans) would say that we’ve exchanged a closure for continuation. There is probably a monad here somewhere, too.
Generators can of course solve a lot of problems that we may want to address with
nonlocal, but it’s true you cannot write them all off just by clever use of
yield statement. For the those rare occasions – when you really, positively, truly need a mutable closure – there are still some options on the board.
The crucial observation is that while the closure in Python 2.x is indeed immutable – you cannot add new variables to it – the objects inside need not be. If you are normally able to change their state, you can do so through captured variables as well. After all, you are still just “reading” those variables; they do not change, even if the objects they point to do.
Hence the solution (or workaround, more accurately) is simple. You need to wrap your value inside a mutable object, and access it – both outside and inside the inner function – through that object only. There are few choices of suitable objects to use here, with
dictionaries being the simplest, built-in options:
If you become fond of this technique, you may want to be more explicit and roll out your own wrapper. It might be something like a
Var class with
set methods, or just a
Finally, there is a variant of the above approach that involves a class rather than function. It is strangely similar to “functor” objects from the old C++, back when it didn’t support proper lambdas and closures. Here it is:
Its main advantage (besides making it a bit clearer what’s going on) is the potential for extracting the class outside of the function – and thus reusing it. In the above example, you would just need to add the
__init__(self, key) method to make the class independent from the enclosing function.
Ironically, though, that would also defeat the whole point: you don’t need a mutable closure if you don’t need a closure at all. Problem solved? ;-)
Today I’d like to present something what I consider rather obvious. Normally I don’t do that, but I’ve had one aspiring Pythonist whom I helped with the trick below, and he marveled at the apparent cleverness of this solution. So I thought it might be useful for someone else, too.
Here’s the deal. In Python, functions can be invoked with keyword arguments, so that argument name appears in the function call. Many good APIs use that feature extensively; database libraries known as ORMs are one typical example:
In this call to
filter_by() we pass the
WHERE clause. By adding more arguments, we can introduce more filters, linked together with the
Suppose, though, that we don’t know the column name beforehand. We just have it stored in some variable, maybe because the query is part of authentication procedure and we support different means for it: e-mail, Facebook user ID, Twitter handle, etc.
However, the keyword arguments in function call must always be written as literal Python identifiers. Which means that we would need to “eval” them somehow, i.e. compute dynamically.
How? Probably best is to construct an ad-hoc dictionary and unpack it with
That’s it. It may not be obvious at first, because normally we only unpack dictionaries that were carefully crafted as local variables, or received as
** works on any dictionary. We are thus perfectly allowed to create one and then unpack it immediately. It doesn’t make much sense in most cases, but this is one of the two instances when it does.
The other situation arises when we know the argument name while writing code, but we cannot use it directly. Python reserves many short, common words with plethora of meanings (computer-scientific or otherwise), so this is not exactly a rare occurrence. You may encounter it when building URLs in Flask:
or parsing HTML with BeautifulSoup:
Strictly speaking, this technique allows you to have completely arbitrary argument names which are not even words. Special handling would be required on both ends of function call, though.
function keyword is essential to becoming an effective JS coder.
So, let’s look into them one by one and see what the
function might really mean.
Any and all code is enclosed within an anonymous
function. It’s not even stored in a
variable; it’s just called immediately so its content is just executed, now.
window in case of web browsers) which is a fragile namespace, easily polluted by defining things directly at the script level.
So in the example above, the
function is used to confine script’s code and all the symbols it defines. But sometimes we obviously want to let some things through, while restricting access to some others – a concept known as encapsulation and exposing an interface.
What we get here is normal JS object but it should be thought of more like a module. It offers some public interface in the form of
getValue functions. But underneath, it also has some internal data stored within a closure: the
value variable. If you know few things about C or C++, you can easily see parallels with header files (.h, .hpp, …) which store declarations that are only implemented in the code files (.c, .cpp).
Or, alternatively, you may draw analogies to C# or Java with their public and private (or protected) members of a class. Incidentally, this leads us to another point…
Let’s assume that the
counter object from the example above is practical enough to be useful in more than one place (a tall order, I know). The DRY principle of course prohibits blatant duplication of code such as this, so we’d like to make the piece more reusable.
Here’s how we typically tackle this problem – still using only vanilla
Pretty straightforward, right? Instead of calling the function on a spot, we keep it around and use to create multiple objects. Hence the function becomes a constructor for them, while the whole mechanism is nothing else but a foundation for object-oriented programming.
It is worth noting why JS has the
this keyword at all. Normally, we would expect it only in those languages which also have the corresponding
class keyword. That’s what C++, Java and C# have taught us: that
this represents the current object of a class when used inside one of its methods. It only makes sense, then, to use
this keyword in a class scope, denoted by the
this even there?
Greeting is technically a function and is defined as one, but semantically it works more like constructor for the
Greeting “class”. As for
this keyword, it refers to the object being created by such a constructor when invoked by
new statement – another familiar construct, by the way. Additionally,
this also appears inside
greet method and does its expected job, allowing access to the
text member of an object that the method was called upon.
So it would seem that everything with
this keyword is actually fine and rather unsurprising. Have we maybe overlooked something here, looking only at half of the picture?…
Well yes, very much so. And not even a half but more like a quarter, with the remaining three parts being significantly less pretty – to say it mildly.
I suppose it this not uncommon to encounter a general situation such as the following. Say you have some well-defined function that performs a transformation of one value into another. It’s not particularly important how lengthy or complicated this function is, only that it takes one parameter and outputs a result. Here’s a somewhat trivial but astonishingly useful example:
Depending on what happens in other parts of your program, you may find yourself applying such function to many different inputs. Then at some point, it is possible that you’ll need to handle lists of those inputs in addition to supporting single values. Query string of URLs, for example, often require such treatment because they may contain more than one value for given key, and web frameworks tend to collate those values into lists of strings.
In those situations, you will typically want to deal just with the list case. This leads to writing a conditional in either the caller code:
or directly inside a particular function. I’m not a big fan of similar solutions because everyone do them differently, and writing the same piece several times is increasingly prone to errors. Quite not incidentally, a mistake is present in the very example above – it shouldn’t be at all hard to spot it.
In any case, repeated application calls for extracting the pattern into something tangible and reusable. What I devised is therefore a general “recursivator”, whose simplified version is given below:
As for usage, I think it’s equally feasible for both on-a-spot calls:
as well as decorating functions to make them recursive permanently. For this, though, it would be wise to turn it into class-based decorator, applying the technique I’ve described previously. This way we could easily extend the solution and tie it to our needs.
But what are the specific ways of doing so? I could think of some, like:
datetimes into ISO-formatted strings.
sets from being turned into lists, as obviously sets are also iterable. In more general version, one could supply a predicate function for deciding whether to recurse or not.
recursiveinto generator for more memory-efficient solution. If we’re lucky to program in Python 3.x, it would be a good excuse to employ the new
yield fromconstruct from 3.3.
One way or another, capturing a particular concept of computation into actual API such as
recursive looks like a good way for making the code more descriptive and robust. Certainly it adheres to one of the statements from Zen: that explicit is better than implicit.
It is common that features dubbed ‘syntactic sugar’ are often fostering novel approaches to programming problems. Python’s decorators are no different here, and this was a topic I touched upon before. Today I’d like to discuss few quirks which are, unfortunately, adding to their complexity in a way that often doesn’t feel necessary.
Let’s start with something easy. Pretend that we have a simple decorator named
@trace, which logs every call of the function it is applied to:
An implementation of such decorator is relatively trivial, as it wraps the decorated function directly. One of the possible variants can be seen below:
That’s pretty cool for starters, but let’s say we want some calls to stand out in the logging output. Perhaps there are functions that we are more interested in than the rest. In other words, we’d like to adjust the priority of log messages that are generated by
This seemingly small change is actually mandating massive conceptual leap in what our decorator really does. It becomes apparent when we de-sugar the
@decorator syntax and look at the plumbing underneath:
Introduction of parameters requires adding a new level of indirection, because it’s the return value of
trace(level=logging.INFO) that does the actual decorating (i.e. transforming given function into another). This might not be obvious at first glance and admittedly, a notion of function that returns a function which takes some other function in order to output a final function might be – ahem – slightly confusing ;-)
But wait! There is just one more thing… When we added the
level argument, we not necessarily wanted to lose the ability to invoke
@trace without it. Yes, it is still possible – but the syntax is rather awkward:
That’s expected –
trace only returns the actual decorator now – but at least slightly annoying. Can we get our pretty syntax back while maintaining the added flexibility of specifying custom arguments? Better yet: can we make
@trace(level) all work at the same time?…
Looks like tough call, but fortunately the answer is positive. Before we delve into details, though, let’s step back and try to somewhat improve the way we are writing our decorators.
Obiekty mają metody. Tak, w tym stwierdzeniu nie należy doszukiwać głębokiego sensu – jest ono po prostu prawdziwe :) Gdy mówimy o metodach obiektów czy też klas, zwykle mamy jednak na myśli tylko jeden ich rodzaj: metody instancyjne. W wielu językach programowania nie jest to aczkolwiek ich jedyny rodzaj – z takim przypadkiem mamy do czynienia chociażby w Pythonie.
Jak zawsze metody instancyjne są domyślnym typem, który tutaj jest dodatkowo zaznaczony obecnością specjalnego parametru –
self – występującego zawsze jako pierwszy argument. To odpowiednik
this z języków C++, C# czy Java i reprezentuje instancję obiektu, na rzecz której wywoływana jest metoda:
Fakt, że musi być on jawnie przekazany, wynika z zasad tworzenia zmiennych w Pythonie. Nie muszą być one jawnie deklarowane. Dlatego też odwołanie do pola obiektu jest zawsze kwalifikowane, gdyż przypisanie do
self._value stworzyłoby po prostu zmienną lokalną.
Istnieją jednak takie metody, które nie operują na konkretnej instancji klasy. Typowo nazywa się je statycznymi. W Pythonie nie posiadają one parametru
self, lecz są opatrzone dekoratorem
Statyczną metodę można wywołać zarówno przy pomocy nazwy klasy (
Counter.format_string()), jak i jej obiektu (
Counter().format_string()), ale w obu przypadkach rezultat będzie ten sam. Technicznie jest to bowiem zwyczajna funkcja umieszczona po prostu w zasięgu klasy zamiast zasięgu globalnego.
Mamy wreszcie trzeci typ, mieszczący się w pewnym sensie pomiędzy dwoma opisanymi powyżej. Nie wydaje mi się jednak, żeby występował on w żadnym innym, popularnym języku. Chodzi o metody klasowe (class methods). Nazywają się tak, bo są wywoływane na rzecz całej klasy (a nie jakiejś jej instancji) i przyjmują ową klasę jako swój pierwszy parametr. (Argument ten jest często nazywany
cls, ale jest to o wiele słabsza konwencja niż ta dotycząca
W celu odróżnienia od innych rodzajów, metody klasowe oznaczone są dekoratorem
Podobnie jak metody statyczne, można je wywoływać na dwa sposoby – przy pomocy klasy lub obiektu – ale w obu przypadkach do
cls trafi wyłącznie klasa. Tutaj akurat będzie to
Counter, lecz w ogólności może to być także klasa pochodna:
Powyższy kod – będący przykładem dodatkowego sposobu inicjalizacji obiektu – to dość typowy przypadek użycia metod klasowych. Korzystanie z nich wymaga aczkolwiek nieco wprawy w operowaniu pojęciami instancji klasy i samej klasy oraz ich poprawnego rozróżniania.
W gruncie rzeczy nie jest to jednak nic trudnego.