Capture stdout/stderr into Log Files

11 October 2014

We want that stdout/stderr messages not only show up on screen but also goes into a log file. This request is very reasonable and seems to be trivial at the first glance: just a "tee" should do the job. But, you cannot "tee" every source line of your script, can you? In this blog, I studied several techniques but found no neat solution.

Redirect stdout/stderr to log file

exec >log_file 2>&1

echo "INFO:blah..."
echo "ERRO:oops" >&2 # still add '>&2' so it is easier to maintain/change
other_commands_or_scripts

This is simple. One issue (amongest others) is that: nothing show up on screen.

Nested subshells with "tee"

(
    (
        (
            echo "INFO:$(date)..."
            echo "ERRO:$(date)"  >&2
            other_commands_scripts
        ) | tee -a log_file     # (in)
    ) 2>&1 1>&3 | tee -a log_file # (mid)
) 3>&1 1>&2                   # (out)

This not only is bewildering but also scrambles the order of output since stdout and stderr are now dealt with in different sub-shells.

During setup
1. In out, fd 3 created and duped to stdout, then stdout redirected to stderr.
2. In mid, stdout go to fd 3 while stderr redirected to stdout and "tee"-ed to log_file.
3. In in, stdout piped to "tee".
Then, messages travel from inside out
1. In in, info messages copied to log_file
2. In mid, info messages sent to fd 3, while error messages sent to fd 1 and copied to log file. to log_file.
3. In out, info messages arrived at fd 3 are print to screen since fd 3 is a dup of the default stdout. Error messages arrived at fd 1 are sent to the default stderr.

More file descriptors

In some cases, it is an advantage that my messages are separated from others' messages.

exec 3>&1                       # 3: original stdout
exec 4>&2                       # 4: original stderr
exec >log_file 2>&1             # stdout/stderr go to log_file

echo "INFO:blah..." >&3
echo "ERRO:oops" >&4
other_commands_or_scripts

Now, my "info" and "error" show up on screen but not log file. Meanwhile, others' output goes to log file but not screen.

"tee" to file descriptors

"tee" is mainly used to copy output to files NOT file descriptors. However, after fiddling with it for sometime, I eventually find a couple of ways to "tee" a file descriptor.

One way is this.

exec 3>&1                       # 3: original stdout
exec 4>&2                       # 4: original stderr
exec >log_file 2>&1             # stdout/stderr go to log_file

echo "INFO:blah..." | tee /dev/fd/3
echo "ERRO:oops" | tee /dev/fd/4
other_commands_or_scripts

Note that for ksh, we have to do it a little different since any "exec" fd bigger than 2 is "private".

(
    exec >log_file 2>&1             # stdout/stderr go to log_file

    echo "INFO:blah..." | tee /dev/fd/3
    echo "ERRO:oops" | tee /dev/fd/4
    other_commands_or_scripts
) 3>&1 4>&2

Now my "info" and "error" go to both screen and log file. But, the stdout/stderr of other commands/scripts still go to log file only.

computer 89