The Subshell Gotcha

2013-05-20 13:13

Many are the quirks of shell scripting. Most are related to confusing syntax, but some come from certain surprising semantics of Bash as a language, as well as the way scripts are executed.
Consider, for example, that you’d like to list files that are within certain size range. This is something you cannot do with ls alone. And while there’s certainly some awk incantation that makes it trivial, let’s assume you’re a rare kind of scripter who actually likes their hacks readable:

  1. #!/bin/sh
  2.  
  3. min=$1
  4. max=$2
  5.  
  6. ls | while read filename; do
  7.     size=$(stat -f %z $filename)
  8.     if [ $size -gt $min ] && [ $size -lt $max ]; then
  9.         echo $filename
  10.     fi
  11. done

So you use an explicit while loop, obtain the file size using stat and compare it to given bounds using a straightforward if statement. Pretty simple code that shouldn’t cause any troubles later on… right?

But as your needs grow, you find that you also want to count how many files fall within your range, and how many do not. Given that you have an explicit if, it appears like a simple addition (in quite literal sense):

  1. matches=0
  2. misses=0
  3. ls | while read filename; do
  4.     size=$(stat -f %z $filename)
  5.     if [ $size -gt $min ] && [ $size -lt $max ]; then
  6.         echo $filename
  7.         ((matches++))
  8.     else
  9.         ((misses++))
  10.     fi
  11. done
  12.  
  13. echo >&2 "$matches matches"
  14. echo >&2 "$misses misses"

Why it doesn’t work, then? Because clearly this is not the output we’re looking for (ls_between is our script here):

  1. $ ls -al
  2. total 25296
  3. drwxrwxr-x  19 xion  staff       646 15 Apr 18:44 .
  4. drwxrwxr-x  15 xion  staff       510 20 May 11:15 ..
  5. -rw-rw-r--   1 xion  staff        16 10 May  2012 hello.py
  6. -rw-rw-r--   1 xion  staff      4005 28 May  2012 keyword_stats.py
  7. -rw-rw-r--   1 xion  staff       218  5 Aug  2012 magical.py
  8. -rw-rw-r--   1 xion  staff     19901 11 May  2012 space_invaders.py
  9. $ ls_between 1024 10241024
  10. keyword_stats.py
  11. space_invaders.py
  12. 0 matches
  13. 0 misses

It seems that neither matches nor misses are counted properly, even though it’s clear from the printed list that everything is fine with our if statement and loop. Wherein lies the problem?

Broken pipe

Surprising as it may be, the issue is within the very first line of our loop:

  1. ls | while read filename; do

The while statement itself is completely fine; however, the way we’re feeding it with input data is not adequate to our needs. We use piping, which is a way to turn the output of one process into the input of another process. A cornerstone of Unix programming and automation, it has one very useful property that allows us to manipulate even very big chunks of data: it doesn’t use any intermediate storage. Piping happens on line-by-line basis between processes running in parallel, and no temporary files are involved.

But why such an efficient method of data processing would be inadequate here? Well, the whole thing about “processes running in parallel” is what gets us here. We establish a pipe between ls and a while loop, which requires the latter to run in a process of its own. That process is simply another instance of the executing shell – subshell – spawned specifically to capture the output of ls, read it and execute our loop’s body.

And this works fine, as long as we don’t try to communicate data back to our main script. The loop lives in a child process, which makes it unable to alter anything in the environment of the parent process – the one executing our complete script. Which means, among other things, that it cannot modify variables from the “outer scope” (such as matches and misses) and have the changes persist after the loop exits. More precisely, it doesn’t even have access to those parent variables: all it gets is a mere local copy of the environment containing them. Sure, it can increment them perfectly well for its own use, but the parent environment will still have those variables set to 0.

Fixing it

The solution to this phenomenon is to avoid creating subshell in the first place and execute the loop in the same process as the rest of our script. For that we need to leave piping aside and find another way of feeding the loop with data.

Fortunately, the while-done syntax permits input redirection, much in the same way as ordinary command execution, i.e. through < operators. The crucial difference is that unlike external commands, the shell builtins – such as while – execute within the running shell’s process. As a result, they have access to the “outer” environment and variables.

You may recall, though, that usage of redirection operator < requires a file:
$ cat <./hello.py print "Hello, world!"[/bash] But we don't want to store the results of ls in a file just to pass them to a loop; that would be very cumbersome, slow and require additional cleanup. Thankfully, in Bash & co. you can use the <() operator to make a temporary “file” (file descriptor, really) that maps to the output of given command:
while read filename; do
size=$(stat -f %z $filename)
if [ $size -gt $min ] && [ $size -lt $max ]; then
echo $filename
((matches++))
else
((misses++))
fi
done < <(ls) # note the space[/bash] There is also a POSIX-complaint alternative: store the output in a variable and redirect from that variable using the dedicated <<< operator:
LS=$(ls)
while read filename; do
size=$(stat -f %z $filename)
if [ $size -gt $min ] && [ $size -lt $max ]; then
echo $filename
((matches++))
else
((misses++))
fi
done <<< “$LS”[/bash] I actually find this variant more readable, probably because the source command is placed before the loop itself. Either one should work in most cases, though.

Tags: , , , ,
Author: Xion, posted under Applications, Programming »


2 comments for post “The Subshell Gotcha”.
  1. Liosan:
    May 20th, 2013 o 13:28

    I think it should be min=$1, max=$2. Your stat command doesn’t work for me either, I had to use ‘stat –printf=”%s” “$filename”‘.

    Anyways, thanks for the weird/nice tip about this feature :) But in this particular case, I would write an “for f in *; do” loop, which is a cooler way to iterate over files.

  2. Xion:
    May 22nd, 2013 o 1:45

    Right, the `ls` was mostly for illustration purposes. If I recall correctly, it was some result of `grep` that I wanted to iterate over when I’ve found about this quirk.

    As for `stat`: that’s portability issue. Turns out `-f` is works in BSD flavors of Unix, while `-printf` does in Linux. I tested this on OS X and didn’t really think about checking Linux man, as I’ve been bitten by any such differences yet. I will know better now :)

Comments are disabled.
 


© 2017 Karol Kuczmarski "Xion". Layout by Urszulka. Powered by WordPress with QuickLaTeX.com.