Redirect to a file STDOUT first and then STDERR

Question

I have a gawk code like this.

#!/usr/bin/gawk -f

1 {
    for (i=0;i<5;i++){
        print $0
        print $0, i > "/dev/stderr"
    }
}

I want to redirect to a file tmp, first the stdout and then the stderr. I tried this:

gawk -f Documents/gawk_script.awk ./file &> tmp

But this call append to the file first stderr. I don't want to divide them into two files, so I'm asking if there's a way to do that.

In ./file there's such a line:

hello
hello
howareyou
well
well

while in tmp file

hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
howareyou
howareyou
howareyou
howareyou
howareyou
well
well
well
well
well
well
well
well
well
well
well
hello 0
hello 1
hello 2
hello 3
hello 4 
hello 0
hello 1
hello 2
hello 3
hello 4
howareyou 0
howareyou 1
howareyou 2
howareyou 3
howareyou 4
well 0
well 1
well 2
well 3
well 4
well 0
well 1
well 2
well 3
well 4

You mention in your question that the file `tmp` contains first `stderr` while in your output the file seems to have first `stdout`. Could you please elaborate what you mean here? — kvantour, Jun 03 '19 at 09:25

Ed Morton · Accepted Answer · 2019-06-01T14:35:57.533

There is no good way* to tell awk or the shell that it must buffer stderr until the tool finishes executing. Keep it simple and just do this:

awk -f script.awk file > out 2>tmp; cat tmp >> out && rm -f tmp

Otherwise you could buffer stderr yourself and print at the end (but this will only work for stderr messages you are manually printing, not messages gawk is generating itself):

{
    for (i=0;i<5;i++){
        print $0
        errs = errs $0 OFS i ORS
    }
}
END {
    printf "%s", errs > "/dev/stderr"
}

and then call as:

awk -f script.awk file > out 2>&1

Of course you don't actually need to use stderr at all if that's all you're doing with it, just print to stdout.

*There may be some arcane incantation you can use to make this happen if the planets align a certain way and/or you have certain tools or a certain shell but just keep it simple as shown above.

score 2 · Answer 2 · answered Jun 03 '19 at 09:45

The problem you are encountering is due to the buffering of the streams stdout and stderr. Both streams have different default buffer-settings. While stdout is line-buffered when writing to a terminal, it is very well buffered when it writes to a stream/pipe/file. The stream stderr on the other hand is always unbuffered. That is why you first encounter the output of stderr and only later the output of stdout in your file tmp. Be aware, however, that the output will be interleaved when you output more lines as all of a sudden the buffer of stdout will be full and written to the file, following again with some output of stderr until the next buffer of stdout is full. The problem is nicely explained in the following page:

http://www.pixelbeat.org/programming/stdio_buffering/

One of the ugly hacks you can apply is the usage of stdbuf to change the buffering of the datastreams of awk:

$ stdbuf -oL -eL awk '{...}' file.txt &> tmp.txt

Here we use stdbuf to set buffering mode of the streams stdout and stderr to line-buffering and thus the output of your code would look like:

hello
hello 1
hello
hello 2
...

If you really want first all output of stdout followed by all output of stderr, you should follow the approach mentioned by Ed Morton.

As an alternative to using `stdbuf`, a call to `fflush()` at the end of every pair of prints within awk can force all of the output to be interleaved as it occurs too. Note that if you're using `print | "cat>&2"` (portable to all awks) to print to stderr instead of `print > "/dev/stderr"` (only works in some awks) then you'll also need to `close("cat>&2")` after every print to avoid `cat` also doing buffering. — Ed Morton, Jun 03 '19 at 17:03

score 1 · Answer 3 · answered Jun 01 '19 at 12:05

1

This is a small example:

I am not sure what you mean, if I understand it...

If you want to redirect stdout to file_out , redirect stderr to file_err , you can do this...

command > file_out 2> file_err

answered Jun 01 '19 at 12:05

Linke

336
1
10

sorry if you don't understand, I'll explain better. Yes, I want to redirect both of them to a file, but the stdout must be redirected before stderr. So in the output file, I will have first the words without number and then those with number. – lmriccardo Jun 01 '19 at 13:16
- So, this is not possible in one step, you need to add some logic to control the text. Suppose your filename is `tmp`. 1. You should create a temporary file. 2. Then redirect stderr to this temporary file 3. After the command is finished, copy and append the stderr in this temporary file to your `tmp` file. 4. Delete this temporary file - If you want to use it multiple times, you need to add another step (add a step between 1-2 above): **cut the number-terminated lines that in `tmp` into the temporary file.** – Linke Jun 01 '19 at 14:31
Okay, So I need to create two files to do that. – lmriccardo Jun 01 '19 at 21:21

score 0 · Answer 4 · answered Dec 28 '20 at 14:17

0

Lit bit old, but if someone still needs it:

echo "some stuff" | awk '{
    for (i=0;i<5;i++){
        print $0;
        # print i on stderr
        system(">&2 echo " i);
    }
}'

answered Dec 28 '20 at 14:17

Carlos Barcellos

526
1
4
9

Redirect to a file STDOUT first and then STDERR

4 Answers4

This is a small example:

Linked