40

I'm working on a bash-script that has to prepare an E-Mail for being sent to a user.

It aggregates some data, which ends up being multiple lines of stuff. For the example stored in $DATA.

Now, after a bit of stfw I found a few things like sed -ei "s/_data_/${DATA}/g" mail.tpl and also sed replace with variable with multiple lines. None of them work.

Now the question is, how do I get sed to replace something with multiple lines of text?

(Alternatives to sed are also welcome!)

anubhava
  • 713,503
  • 59
  • 514
  • 593
Cobra_Fast
  • 14,992
  • 8
  • 56
  • 101
  • 1
    possible duplicate of [sed replace with variable with multiple lines](http://stackoverflow.com/questions/6684487/sed-replace-with-variable-with-multiple-lines) – Zsolt Botykai Apr 11 '12 at 14:06
  • 2
    @ZsoltBotykai You dont say? :D I even mentioned it in my question... – Cobra_Fast Apr 11 '12 at 14:07
  • As an aside, don't use uppercase for your private shell variables. – tripleee Mar 11 '18 at 12:23
  • @tripleee why not? – Cobra_Fast Mar 11 '18 at 15:44
  • 1
    Because that's how you avoid inadvertent clashes with reserved variables, which are uppercase. There's no consensus on what *exactly* this means but your script's private variables are definitely not "system variables" as intended in this clause in POSIX. See https://stackoverflow.com/q/673055/874188 – tripleee Mar 11 '18 at 15:54
  • @Cobra_Fast any chance you could consider accepting a different answer? – tripleee Mar 11 '18 at 15:55

9 Answers9

28

You can do this with AWK using variable substitution. We can set a variable in AWK using -v, and then use AWK's gsub function to substitute all occurrences of a regular expression with that variable.

For example, if the file test has the following contents ...

foo
bar
blah _data_and_data_
foo
_data_ foobar _data_ again

... and the Bash variable $DATA is ...

1
2
3
4
5

... then awk -v r=$DATA '{gsub(/_data_/,r)}1' test replaces all occurrences of the regular expression _data_ in the file test with the contents of $DATA, resulting in the following:

foo
bar
blah 1
2
3
4
5and1
2
3
4
5
foo
1
2
3
4
5 foobar 1
2
3
4
5 again
Jivan Pal
  • 140
  • 8
Kent
  • 181,427
  • 30
  • 222
  • 283
16

I would suggest simply replacing sed with perl command like this:

perl -i.bak -pe 's/_data_/$ENV{"DATA"}/g' mail.tpl
anubhava
  • 713,503
  • 59
  • 514
  • 593
  • 2
    This is the only that worked for me on OSX. All the other sed and awk based approches failed fo me. – Cédric Vidal Dec 01 '17 at 13:45
  • 4
    This will fail in unsettling ways if `DATA` contains a slash. But you can `export DATA` and access it from inside Perl as `$ENV{"DATA"}` without getting code and data mixed up. So `'s/_data_/$ENV{"DATA"}/g'` simply. – tripleee Mar 11 '18 at 12:22
  • 1
    @anubhava and @tripleee, you saved my day. This is the only solution that worked for me: neither `sed` nor `awk` could do the trick. Thanks. – Dmytro Titov Jul 04 '19 at 14:37
15

If you build your multiple line text with "\n"s, this will work with a simple sed command as:

DATA=`echo ${DATA} | tr '\n' "\\n"`
#now, DATA="line1\nline2\nline3"
sed "s/_data_/${DATA}/" mail.tpl
ring bearer
  • 19,685
  • 7
  • 57
  • 70
  • 5
    I'm getting `sed: -e expression #1, char 32: unterminated 's' command` from that method. – Cobra_Fast Apr 11 '12 at 14:20
  • 1
    This is not entirely portable; not all `sed` dialects accept the `\n` digraph to represent a literal newline. In some other dialects, you will probably need to use a backslash before a literal newline. Other answers on this page show you how to do that in more detail. – tripleee Jul 06 '17 at 04:11
  • 8
    Looking at this again, I don't think there is *any* version of `tr` which can replace one character with two characters; it's simply the wrong tool for this job. The newlines are lost by accident because you incorrectly fail to quote the argument to `echo`, but this has multiple other undesirable side effects; and the `tr` simply does nothing at all here. **This should not be the accepted answer.** – tripleee Mar 11 '18 at 12:26
  • Yes does not work for me either. Try [this solution (below)](https://stackoverflow.com/questions/10107459/replace-a-word-with-multiple-lines-using-sed/#22901380) – TechupBusiness Apr 15 '19 at 15:26
  • 1
    Given that the shell is Bash, then there's no need for the (broken) `tr` invocation - [just use `${DATA//$'\n'/\\n}`](/a/39269241/4850040). – Toby Speight Aug 14 '19 at 13:08
11

I tried it and sed 's/pattern/\na\nb\nc/g' but it does not work on all systems. What does work is putting a \ followed by a newline in the replace pattern, like this:

sed 's/pattern/a\
b\
c/g'

This appends a line containing b and a line containing c when the pattern is seen.

To put it in a variable, use double backslashes:

export DATA="\\
a\\
b\\
c"

and then:

sed "s/pattern/${DATA}/g"

Note the double quotes.

DᴀʀᴛʜVᴀᴅᴇʀ
  • 5,839
  • 15
  • 57
  • 101
Albert Veli
  • 1,291
  • 7
  • 9
11

ring bearer's answer didn't work for me; I think the usage of tr there is wrong, and the way it's written, it simply strips away newlines by use of echo.

Instead, I used sed. I used code from another answer to replace newlines (credit: Zsolt Botykai). I also expected some dollar signs ($) in my input so I took care of that too. You might need to add other input handling. Note the use of double quotes in echo to preserve newlines.

DATA="$(cat whatever)"
ESCAPED_DATA="$(echo "${DATA}" | sed ':a;N;$!ba;s/\n/\\n/g' | sed 's/\$/\\$/g')"

Then you can use ${ESCAPED_DATA} in sed:

cat input | sed 's/one liner/'"${ESCAPED_DATA}"'/' > output

Just thought I'd share.

Community
  • 1
  • 1
Yuval
  • 2,871
  • 28
  • 41
  • This works well also when multi-line strings have been built in a `bash` script with `$'\n'` construct (the `cat` in the beginning is, of course, then not needed since the string is already in a variable). – Ville Jul 30 '17 at 21:43
8

Echo variable into temporary text file.

Insert text file into mail.tpl and delete data from mail.tpl

echo ${DATA} > temp.txt    
sed -i -e "/_data_/r temp.txt" -e "//d" mail.tpl
Ben Pingilley
  • 679
  • 1
  • 6
  • 13
4

Escaping all the newlines with a \ (except the last one) worked for me. The last newline must not be escaped not to break the s command.

Example :

DATA="a
b
c"

ESCAPED=$(echo "${DATA}" | sed '$!s@$@\\@g')
echo "${ESCAPED}" 
a\
b\
c

sed "s/pattern/${ESCAPED}/" file
mgraff
  • 41
  • 1
2

You can put your data in a temp file and run:

$ sed '/_data_/r DATA_FILE' mail.tpl | sed '/_data_/d'> temp; mv temp mail.tpl
Mihai
  • 2,017
  • 2
  • 13
  • 16
1

Not sure if you have tried to put "\n" in the replace part

sed 's/[pattern]/\
[line 1]\n\
[line 2]\n\
[line n]\n\
/g' mail.tpl

The first line has /\ for readibility reasons. Each line after that is a stand-alone line like you would find in a text editor. Last line is stand-alone, once again for readability reasons. You can make all of this one line if needed. Works on Debian Jessie when I tested it.

lsu_guy
  • 1,329
  • 13
  • 12