Output Redirection With Bash

RedirectionWhenever you work with bash there are three file descriptors that are always in play and that you need to be aware of. These are, STDOUT, STDERR and STDIN which stand for standard output, standard error and standard input. Normally when programs are executed, unless they explicitly write output to a file, they will output to either standard output or standard error, which means the output will go to the terminal and you will see it on the screen (since both STDOUT and STDERR go to the screen by default). STDIN is where your program will get input data from if you prompt for input from the user. The user enters some data on the screen and your program or script will read it (I won’t say too much more about STDIN).

Of course STDOUT and STDERR are concepts, the actual file descriptors themselves are numbers. When you’re using bash, 1 will be the file descriptor that stands for STDOUT and 2 will be the file descriptor that stands for STDERR. So when you want to redirect the output of a program or script to a file rather than going to the screen you need to change where STDOUT or STDERR (or both) are pointing. To do this you can do the following:

ls -al 1>file1.txt 2>file2.txt

You use the > ‘operator’ to redirect where STDOUT and STDERR are going to write their output by giving the > ‘operator’ a file descriptor and the file you want the output to be written to. In this case nothing will appear on the screen but two new files will be created in the current directory, file1.txt will have the contents of STDOUT and file2.txt the contents of STDERR (of course unless an error actually occurred file2.txt will be empty). If you don’t specify any number e.g.:

ls -al > file1.txt

bash assumes you want to redirect STDOUT, so STDOUT will go to the file you supply, while STDERR will still go to the screen.

But what if you want to redirect both standard output and standard error to the same file. This is easily possible and simply builds on the previous concept. You may have seen the following construct before:

ls -al > file1.txt 2>&1

What this actually means is as follows. We first redirect standard output to a particular file, remember when we don’t specify a number, bash assumes we want to redirect STDOUT:

ls -al > file1.txt

The last bit of the command (2>&1) literally means:

  • take STDERR (i.e. file descriptor 2)
  • and redirect it (i.e. > )
  • to the same place where STDOUT is pointing (&1)

The &1 in the last bit is a little tricky. You may think of the & as a sort-of reference and the 1 is of course STDOUT. So, &1 means – ‘the place that STDOUT is currently referencing‘. Since we’ve previously redirected STDOUT to a file this will redirect STDERR to that same file. This is actually the old-school way of redirecting both STDOUT and STDERR to the same place.  There is a shortcut, you simply need to do the following:

ls -al >& file1.txt

which will achieve the same result.

A couple of interesting points to note here.

1. You can easily turn this redirection around and redirect STDERR first and then point STDOUT to the same place, but this looks a little more clunky since bash can’t assume anything and so we need to specify everything explicitly:

ls -al 2> file1.txt 1>&2

As you can see we redirect STDERR (2) to a file first and then use the same construct as before but swap the numbers around. This is good to know and demonstrates that we understand what is going on, but we probably wouldn’t do this in the wild. Instead use the shortcut version I described above.

2. When you redirect both standard output and standard error to the same file, you may get some unexpected results. This is due to the fact that STDOUT is a buffered stream while STDERR is always unbuffered. This means that every character of STDERR is written as soon as it is available while STDOUT writes stuff in batches. When both STDOUT and STDERR are going to the same file you may see error messages appear sooner than you would have expected them in relation to the actual output of your program or script. It isn’t anything to be alarmed about but is simply a side-effect of buffered vs. unbuffered streams, you just need to keep it in mind.

That’s all there is to know about bash redirection, always good to be comfortable with the basics. If I’ve missed anything, feel free to fill in the blanks (i.e. leave a comment :)).

For more tips and opinions on software development, process and people subscribe to skorks.com today.

Image by Phillie Casablanca

  • Thura

    I just use ls -al &> file.txt to redirect both stdout & stderr …

    • http://www.skorks.com Alan Skorkin

      Yeah, that’s the best way.

      • Jen

        Can you explain this version? I can’t quite get it, 1 (stdout) is assumed and I see the & for referencing but where is 2 and how does it work? Thanks, btw, great article with understandable explanations.

  • Tiago

    Thanks for the short but clear explanation.
    In my case, I need just stderr printed, and discard stdout. I’ve found the following way of doing that:
    ls -al 2>&1 1>/dev/null

    Is there a better way? Or, even if not better, other simple way of doing the same thing?

    • http://www.skorks.com Alan Skorkin

      If you simply want stderr printed to the screen, then that is what the shell will do by default, so if you need to discard stdout but print stderr to the console, all you have to do is redirect stdout to /dev/null:

      ls -al > /dev/null

      There is no need to to point stderr to stdout (2>&1) as by default they both go to the console. Of course the above command will print nothing as it will succeed and output will go to /dev/null. But if you cause the command to fail you will get output on the console.

  • Ray

    Hi. Your explanation was very clear and to the point. Thank you for this. I also like to ask you to add some light to the redirection with pipe and exec commands if possible.

    Ray

  • Maxim

    First, let me thank you for this bit of info. It’s one of the best articulated information on redirection I’ve found around. So thank you.

    However, it still does not answer one of the case I’m trying to implement : direct only stderr to a file, and print both stderr and stdout to console.

    Would you happen to know a simple way to do this? Thanks in advance!

    • http://www.skorks.com Alan Skorkin

      Glad you found it useful. What you want to do is not too difficult, infact I’ve written about something similar before. You can check out my other post on redirection, but here is basically what you want to do.

      The tee command is your friend (I explain it the post I linked to above). What you have to do is, swap stdout and stderr around so that you can pipe stderr to tee, that’s pretty much it. Here is a sample command:

      ls -al 3>&1 1>&2 2>&3- | tee blah.txt

      If you run that one you will get your output on the screen but nothing in the file, since no error was produced. But if you run this:

      ls -garbage 3>&1 1>&2 2>&3- | tee blah.txt

      You will get your error output both on the screen and in the file, just as you wanted. The reason you have to swap stdout and stderr around is because you can only use pipes with stdout and you need to pipe your output to tee to get it in a file and on the screen. Hope this helps.

      • Maxim

        Thanks, Alan!

        Your explanation did not only enabled my script to do exactly what I wanted, but I think I now fully understand how bash’s file descriptors can be used. :)

        Kudos! Have a nice week.

        • Maxim

          Just a small observation : as I’m using DASH instead of BASH (for better portability and POSIX compliance) I had to close the file descriptor #3 explicitly with 3>&- instead of the more compact 2>&3- form that BASH allowed. So it would now looks like : 3>&1 1>&2 2>&3 3>&- in my DASH compatible script.

  • Vamsi

    Thank you. This was very informative and precise as well.