How to fix stdio buffering

It’s a common problem. You write some shell command like:

$ tail -f /var/log/foo | egrep -v 'some|stuff' | sed | awk

and wonder why nothing is printed, even though you know some text has matched. The problem is that stdio is being buffered, and there’s a very good write-up of the problem here so I won’t repeat the technical background.

What I will provide though is how to fix it for common cases.

stdbuf

stdbuf is part of GNU coreutils, and is essentially an LD_PRELOAD hack which calls setvbuf() for an application. Thus it is a generic solution to the problem and can be used to fix most applications. Usage looks like this:

$ tail -f /var/log/foo | stdbuf -o0 app ...

which will disable output buffering for app, assuming it does not do something itself to reverse thesetvbuf() call. An example of a misbehaving application is mawk, below.

awk

GNU awk needs no modifications, that is it does not buffer when there is no controlling tty.

mawk however (the default awk in Debian/Ubuntu and possibly others) buffers output, and also does not seem to work with stdbuf. It does however provide a -Winteractive option which will turn off buffering.

$ tail -f /var/log/foo | gawk

or

$ tail -f /var/log/foo | mawk -Winteractive

sed

GNU sed provides the -u option which calls fflush(), thereby providing unbuffered output. You can also use stdbuf as above.

$ tail -f /var/log/foo | sed -u

or

$ tail -f /var/log/foo | stdbuf -o0 sed

grep

Similar to sed, GNU grep provides a specific option, --line-buffered, to disable buffering, or again you can use stdbuf.

$ tail -f /var/log/foo | grep --line-buffered

or

$ tail -f /var/log/foo | stdbuf -o0 grep

This article is copied from http://www.perkin.org.uk/posts/how-to-fix-stdio-buffering.html.