[Utility Post] STDOUT, STDERR, & Redirection for the Perplexed & Curious

(Psst — if you’re busy and want a cheat-sheet see here: https://github.com/valgaze/redirection-fun/blob/master/cheatsheet.md)

Put simply, “redirection” is a method to save output and errors to a file (or suppress them entirely) instead of their default behavior of displaying on the screen. This can be useful because it may sometimes be desirable to store output/error logs for later analysis instead of letting them race by on the terminal. Redirection is also useful when using long-running or noisy tools.

If you want to run npm install, for example, and save the output to one file and any errors to a different file, this is all it takes:

npm install >npm-log.txt 2>npm-errors.txt

[If you’re curious about redirection & npm, see /example directory of the companion repo: https://github.com/valgaze/redirection-fun]

Or imagine there exists some tool called “myapp.” If you knew nothing about “myapp” — not knowing its purpose, functionality or output — is there anything you can say with reasonable certainty about the following command?

$ /dev/bin/myapp >logfile.log 2>&1

Probably not much, but if myapp behaves like most other command line programs (and you’re familiar with redirection!), you could at least say the following:

Its Standard Output (if any) will be written to logfile.log
Its Standard Error (if any) will also be written to logfile.log (mixed in with Standard Output)

This note will explain what that means, why this is the case, and detail some other examples of how to perform “redirection” in the terminal.

If you want to follow along with the examples, type the following in your command prompt:

git clone https://github.com/valgaze/redirection-fun redirection-fun && cd $_

Streams & Redirection

Whenever you run a command in the terminal, you have access to three ‘streams’: Standard Input (stdin), Standard Output (stdout), and Standard Error (stderr)

In short, whenever you see “good news” from running a command that is probably coming from its stdout (Standard Output) stream and whenever you see any “bad news” (errors or warnings) odds are it is coming from stderr (Standard Error.)

By default these so-called “standard streams” will always display right in the terminal once a command is run. But since the streams are standardized and plaintext (ie human-readable) you can in fact “redirect” them elsewhere using a pretty simple but powerful syntax.

For instance, when you use the “ls” command you will see its output (ie a listing of files) display right in your terminal as expected:

😎 ~/redirection-fun ls

1.json 3.json 5.json README.md2.json 4.json 6.json example

That’s because the STDOUT stream (Standard Output) is a list of files and the stdout stream selects the terminal as its destination by default.

If we use redirection, however, instead of displaying in the terminal the output can instead be redirected to a file_:_

😎 ~/redirection-fun ls 1>output.txt

If you inspect output.txt you’ll discover that it now contains that Standard Output from the ls command we saw earlier:

😎 ~/redirection-fun cat output.txt1.json2.json3.json4.json5.json6.jsonREADME.mdexampleoutput.txt

In other words, rather than display in the terminal, stdout was “redirected” to output.txt

The procedure is the same for errors. If you type the ls command and ask it to list files in a directory that does not exist (here a non-existent directory called “bongo”) you will get an error:

😎 ~/redirection-fun ls bongo/ls: bongo/: No such file or directory

The error above is from STDERR and like STDOUT by default its destination is the terminal. We can use redirection to save it to a file named “errors.txt” instead:

😎 ~/redirection-fun ls bongo/ 2>errors.txt

If you inspect errors.txt, guess what you’ll find?

😎 ~/redirection-fun cat errors.txtls: bongo/: No such file or directory

This is what redirection is all about — redirecting data from its default destination (the terminal) and moving it somewhere else. As you’ll see below, you can send the data from stdout and stderr to a single file, separate files, or discard it entirely.

Standard Output

If you’re following along at home (https://github.com/valgaze/redirection-fun), try the following inside the project’s main directory:

😎 ~/redirection-fun ls 1>output.txt

You should see no output on your screen and if you inspect output.txt it will be filled with the stdout from the ls command (ie what would have displayed in your terminal.)

Important: The special number 1 is called a File Descriptor and here it represents Standard Output and can be safely dropped when referring to stdout. All of the following commands are identical tols 1>output.txt:

ls 1>output.txtls 1> output.txtls > output.txtls >output.txt

For the rest of this note, we’ll follow the syntax of last item in the list above (no file-descriptor and no space for stdout.) The command above will create the file output.txt if it does not already exist and will overwrite it if it does exist. Note that if you are redirecting to a file to a specific subdirectory (ex ls > /usr/abcd/output.txt) that subdirectory must already exist otherwise the command will fail.

If you want to append to an existing file (not overwrite) stdout, you would do the following (two arrows):

😎 ~/redirection-fun ls >>output.txt

If you inspect output.txt you will see that since we ran the command twice, the standard output is doubled:

😎 ~/redirection-fun cat output.txt1.json2.json3.json4.json5.json6.jsonREADME.mdexampleoutput.txt1.json2.json3.json4.json5.json6.jsonREADME.mdexampleoutput.txt

We know how to write stdout to a file and append it to an existing file but what if we want to get rid of stdout entirely? How does one run a command completely free of any (standard) output?

To discard all stdout data, you should redirect the command to the “null” device stream located on the path /dev/null

😎 ~/redirection-fun ls >/dev/null

For an “informational” command like ls, redirecting to /dev/null does not make a lot of sense but for other commands (especially those with long-running and/or noisy output) redirecting to /dev/null is the best way to discard stream data.

To recap, all you need to know to redirect stdout is:

ls >/dev/tty # redirects to terminal, default behavior

ls >output.txt # redirects stdout to a file, here output.txt

ls >>output.txt # appends stdout to a file, here output.txt

ls >/dev/null # discards stdout entirely by redirecting it to /dev/null

Standard Error

Standard Error (stderr) is similar to stdout but if stdout is most of the “good news” then stderr is all of the “bad news.” Most of the syntax is identical, however, the only change is that stderr’s File Descriptor (the number 2) cannot be dropped.

To redirect errors to a file named errors.txt, simply run:

😎 ~/redirection-fun ls bongo 2>errors.txt

If you inspect errors.txt you’ll get your error:

😎 ~/redirection-fun cat errors.txtls: bongo: No such file or directory

It’s the same story as appending to an existing file:

😎 ~/redirection-fun ls bongo 2>>errors.txt

If you inspect errors.txt after appending you’ll see multiple entries as expected:

😎 ~/redirection-fun cat errors.txtls: bongo: No such file or directoryls: bongo: No such file or directory

Just like with stdout if you want to discard stderr entirely (neither display on screen nor write to a file), simply redirect stderr to the null device:

😎 ~/redirection-fun ls bongo 2>/dev/null

Redirecting Standard Error, besides the mandatory File Descriptor (ie the number 2), will look very similar to redirecting Standard Output:

ls 2>stderr.txt # Write stderr to file

ls 2>>stderr.txt # Append errors to file

ls 2>/dev/null # Discard stderr entirely

Combining Stdout & Standard Error

Oftentimes you will want stdout and stderr to be captured at the same time.

Simple case: separate files (overwrite)

😎 ~/redirection-fun ls >output.txt 2>errors.txt

Here stdout is redirected to output.txt and stderr is redirected to errors.txt

Simple case: separate files (append, don’t overwrite)

😎 ~/redirection-fun ls >>output.txt 2>>errors.txt

Just like above, stdout and stderr are appended to end of the their new destination files.

Trickier case: SAME file (overwrite)

If you want both stdout AND stderr to share the same file there are a number of approaches. One approach involves “duplicating” the stream using “&num” syntax. You first redirect stdout and then redirect stderr to the same destination by using the & symbol and the number to instruct stderr to go wherever stdout is going, for instance:

😎 ~/redirection-fun ls >shared-output.txt 2>&1

Here stdout (1) is redirected to output.txt and stderr (2) is redirected to wherever stdout is going (in this case output.txt)

Note that the following will have an identical outcome to above:

😎 ~/redirection-fun ls 2>shared-output.txt >&2

Here stderr (2) is first redirected to shared-output.txt first and then stdout (1) is redirected to stderr’s destination so it too goes to shared-output.txt

If you want to append stdout and stderr to the same file using this syntax you would perform either of the following:

😎 ~/redirection-fun ls >>shared-output.txt 2>&1

😎 ~/redirection-fun ls 2>>shared-output.txt >&2

In the first command, stdout (1) is being redirected in append mode (two arrows) to shared-output.txt and stderr (2) follows stdout so it too is being redirected to shared-output.txt. In the second command the logic is exactly the same, however, stdout (1) follows stderr’s (2) lead instead.

A “portable” way to redirect both stdout and sterr to the null device (ie put them in a trash can), you would write something like the following:

😎 ~/redirection-fun ls >/dev/null 2>&1

Danger: Order Matters!

The order matters when when combining stdout and stderr to the same destination order. For example, if we intended both stdout and stderr to both share a file named shared-output-2.txt, the following commands would fail:

😎 ~/redirection-fun ls >&2 2>shared-output-2.txt

# This first command would fail and in fact displays stdout on the screen when run. The reason is that since order matters, stdout (1) copies stderr’s initial default destination (the terminal or /dev/tty) and then stderr (2) is redirected to shared-output-2.txt. Stdout was copying stderr's first destination (the terminal, it's default) and not it's final redirected destination (the file shared-output2.txt)

😎 ~/redirection-fun ls 2>&1 >shared-output-2.txt

# This command would not redirect stderr to shared-output-2.txt. Going in order, stderr (2) is redirected to stdout's initial destination (/dev/tty or terminal) and only later is stdout (1) redirected to shared-output-2.txt

To recap, to redirect both stdout and stderr streams…

Different destination:

😎 ~/redirection-fun ls >output.txt 2>errors.txt😎 ~/redirection-fun ls >>output.txt 2>>errors.txt # (no overwrite)

Same destination:

😎 ~/redirection-fun ls >shared-output.txt 2>&1😎 ~/redirection-fun ls 2>shared-output.txt 1>&2😎 ~/redirection-fun ls >>shared-output.txt 2>&1 # (no overwrite)😎 ~/redirection-fun ls >/dev/null 2>&1

Standard Input (stdin) and piping

Standard Input (stdin)

Standard Input (stdin), like stdout and stderr, is an abstraction — the details of how the data is generated or where it comes from does not matter. For example, usually the stdin for a program would come from a source like a keyboard, but redirecting stdin makes it possible for a command’s input to come from a file instead:

command < file

For instance if you wanted to list files containing “json” in their name, you’d run these two commands:

ls >files.txt # stdout to files.txtgrep json < files.txt # stdin to grep is from files.txt

Here ls lists the current files/folders and instead of stdout displaying on the terminal, stdout is redirected to files.txt. In the next command we run grep json and we redirect stdin to grep. We didn’t redirect grep’s stdout so when it runs it will display a list of files (if any exist) with json in their filename in the terminal.

Piping

Piping is very similar to redirecting to stdout but instead of writing to a file, piping only redirects stdout as the stdin to another program.

Instead of writing above with two commands, we accomplish the same thing by piping:

ls | grep json

😎 ~/redirection-fun ls grep | json1.json2.json3.json4.json5.json6.json

The ls command lists files and then its stdout is then “piped” over as stdingrep json whose stdout is not being redirected so it displays in the terminal.

Fun Example: A bootable Live-CD from a tweet

Security engineer Alok Menghrajan crafted a command small enough to squeeze into 280 characters that generates a bootable live-cd for a retro computer game:

The command itself a bit messy but for a moment if you take for granted Alok’s brilliant Perl trick, it’s just piping and redirecting: the Perl command output is piped to a command to encode it in base64 and then the STDOUT of thatbase64 operation is redirected to the file cd.iso— pretty nifty:

$ perl -E "ALOKS_PERL_TRICK" | base 64 -D >cd.iso

That should do it, happy redirecting!