Posts tagged ‘with’

Working within Temporary Directory

2012-03-24 15:35

Few days ago I needed to write a script which was supposed to run inside a temporary directory. The exact matter was about deployment from an ad hoc Git repository, and it’s something that I may describe in more detail later on. Today, however, I wanted to focus on its small part: a one that (I think) has neatly captured the notion of executing something within a non-persistent, working directory. Because it’s a very general technique, I suppose quite a few readers may find it pretty useful.

Obtaining a temporary file or even directory shouldn’t be a terribly complicated thing – and indeed, it’s very easy in case of Python. We have a standard tempfile module here and it serves our needs pretty well in this regard. For one, it has the mkdtemp function which creates a temporary directory and returns path to it:

  1. temp_dir = tempfile.mkdtemp()

That’s what it does. What it doesn’t do is e.g ensuring a proper cleanup once the directory is not needed anymore. This is especially important on Windows where the equivalent of /tmp is not wiped out at boot time.
We also wanted our fresh temp directory to be set as the program’s working one (PWD), and obviously this is also something we need to manually take care of. To combine those two needs, I think the best solution is to employ a context manager.

Context manager is basically a fancy name for an object that the with statement can be applied upon. You may recall that some time ago I wrote about interesting use cases for the with construct. This one could also qualify as such, but the principles are very typical. It’s about introducing a scope where some resource (here: a temporary directory) remains accessible as long as we’re inside it. Once we leave the with block, it is cleaned up – just like file handles, network sockets, concurrent locks and plenty of other similar objects.

But while semantics are pretty clear, there are of course several ways to do this syntactically. I took this opportunity to try out the supposedly simplest one which I learned recently on local Python community meet-up: the contextlib library. It includes the contextmanager decorator: a simple and clever way to write with-enabled objects as simple functions. It is based on particular usage of yield statement which makes it very interesting even by itself.

So without further ado, let’s look at the final solution I wanted to present:

  1. import os
  2. import shutil
  3. import tempfile
  4. from contextlib import contextmanager
  6. @contextmanager
  7. def temp_directory(*args, **kwargs):
  8.     """Allows the program to operate inside temporary directory.
  9.    Sets the app's working dir automatically and restores it
  10.    to original one upon existing the `with` clause.
  11.    """
  12.     orig_workdir = os.getcwd()
  13.     temp_workdir = tempfile.mkdtemp(*args, **kwargs)
  14.     os.chdir(temp_workdir)
  16.     yield temp_workdir
  18.     os.chdir(orig_workdir)
  19.     shutil.rmtree(temp_workdir)

As we can see, yield divides this function into two parts: setup and cleanup. Setup will be executed when we enter the with block, while cleanup will run when we’re about to exit it. By the way, this scheme of multiple entry and exit points in one function is typically referred to as coroutine, and it allows for several very intriguing techniques of smart computation.

Usage of temp_directory function is pretty obvious, I’d say. Here’s a simplified excerpt of the Git-based deployment script that I used it in:

  1. import subprocess
  2. shell = lambda cmd:, shell=True)
  4. orig_repo = os.getcwd()
  5. with temp_directory():
  6.     shell('git clone --shared %s .' % orig_repo)
  7.     shell('./build')
  8.     shell('git add -f ' + build_products)
  9.     shell('git commit -m "%s"' % message)
  10.     shell('git push %s master' % deploy_remote)

Note how the meaning of '.' (current directory) shifts depending on whether we’re inside or outside the with block. Users of Fabric (Python- and SSH-based remote administration tool) will find this very similar to its cd context manager. The main difference is of course that directory we’re cd-ing to is not a predetermined one, and that it will disappear once we’re done with it.

Tags: , , ,
Author: Xion, posted under Programming » Comments Off on Working within Temporary Directory

On Clever Usage of Python ‘with’ Clauses

2012-01-21 20:21

As the Python docs state, with statement is used to encapsulate common patterns of tryexceptfinally constructs. It is most often associated with managing external resources in order to ensure that they are properly freed, released, closed, disconnected from, or otherwise cleaned up. While at times it is sufficient to just write the finally block directly, repeated occurrences ask for using this language goodness more consciously, including writing our own context managers for specialized needs.

Those managers – basically a with-enabled, helper objects – are strikingly similar to small local objects involved in the RAII technique from C++. The acronym expands to Resource Acquisition Is Initialization, further emphasizing the resource management part of this pattern. But let us not be constrained by that notion: the usage space of with is much wider.


© 2017 Karol Kuczmarski "Xion". Layout by Urszulka. Powered by WordPress with