Using sed to delete all lines between two matching patterns

Question

I have a file something like:

# ID 1
blah blah
blah blah
$ description 1
blah blah
# ID 2
blah
$ description 2
blah blah
blah blah

How can I use a sed command to delete all lines between the # and $ line? So the result will become:

# ID 1
$ description 1
blah blah
# ID 2
$ description 2
blah blah
blah blah

Can you please kindly give an explanation as well?

score 91 · Accepted Answer · edited Sep 27 '17 at 10:05

91

Use this sed command to achieve that:

sed '/^#/,/^\$/{/^#/!{/^\$/!d}}' file.txt

Mac users (to prevent extra characters at the end of d command error) need to add semicolons before the closing brackets

sed '/^#/,/^\$/{/^#/!{/^\$/!d;};}' file.txt

OUTPUT

# ID 1
$ description 1
blah blah
# ID 2
$ description 2
blah blah
blah blah

Explanation:

/^#/,/^\$/ will match all the text between lines starting with # to lines starting with $. ^ is used for start of line character. $ is a special character so needs to be escaped.
/^#/! means do following if start of line is not #
/^$/! means do following if start of line is not $
d means delete

So overall it is first matching all the lines from ^# to ^\$ then from those matched lines finding lines that don't match ^# and don't match ^\$ and deleting them using d.

edited Sep 27 '17 at 10:05

jackscorrow

660
1
9
26

answered Jun 09 '11 at 03:59

anubhava

713,503
59
514
593

17

For Mac users: To prevent `extra characters at the end of d command` error you need to add semicolons before the closing brackets `sed '/^#/,/^\$/{/^#/!{/^\$/!d;};}' file.txt` – AvL Nov 12 '13 at 23:02
7

Is that the reason to downvote? If you like an answer then upvote it. Downvote is usually for some answer that doesn't solve OP's problem and since OP has accepted this answer it means it worked for OP. Isn't it? – anubhava Feb 18 '20 at 10:44
1

How would you do that if you want to include the # and $ lines for deletion? If you want to find $ at the end of a line, you can do $\$, – Timo Oct 30 '20 at 18:57
2

Then just use: `sed '/^#/,/^\$/d' file` – anubhava Oct 30 '20 at 19:34
I used `sed '/^====/,/^>>>>/d' file-with-git-merge-conflicts.xml > file-ok.xml` to remove git merge conflict lines between '=======' and '>>>>>>> branch-name, and `sed '/^<<< – bcag2 Apr 06 '22 at 07:40

score 56 · Answer 2 · answered Jan 24 '13 at 02:22

56

$ cat test
1
start
2
end
3
$ sed -n '1,/start/p;/end/,$p' test
1
start
end
3
$ sed '/start/,/end/d' test
1
3

answered Jan 24 '13 at 02:22

Lri

25,332
8
81
78

2

The speed at which this works on 300mb files is impressive. I'm talking split second on a SSD. – Ray Foss Apr 28 '15 at 19:15
1

I was a little confused since I'm not familiar with the sed syntax. It was not clear that the first and second sed commands did not have a dependency - i.e. the difference between the two is whether you want to preserve the match token or not. Until I tested it, I had assumed the first command removed everything between the tokens and the second removed the tokens themselves. If you're trying to strip out a chunk between tokens you only need to use the second command. – lukevp Feb 24 '19 at 21:56
No idea why, but `'1,/start/p;/end/,$p'` completely screwed up my workflow, as I was relying on this working. It does not work at all for me. – Akito Jan 31 '20 at 14:09
https://github.com/theAkito/akito-libbash/blob/dd91364083f13d1132d68489172bbce664b9c9c0/setup.bash#L37 is the line in question. Did I miss something? Because to me it looks like it is exactly as you have shown in your answer @Lri. – Akito Jan 31 '20 at 14:11
2

The solution that actually works is the following: `sed '/PATTERN-1/,/PATTERN-2/{//!d}' input.txt` – Akito Jan 31 '20 at 14:20
Thank you @Akito this works for me too in Fish shell – eyeseaevan May 21 '21 at 19:30

score 21 · Answer 3 · edited Sep 03 '18 at 23:45

In general form, if you have a file with contents of form abcde, where section a precedes pattern b, then section c precedes pattern d, then section e follows, and you apply the following sed commands, you get the following results.

In this demonstration, the output is represented by => abcde, where the letters show which sections would be in the output. Thus, ae shows an output of only sections a and e, ace would be sections a, c, and e, etc.

Note that if b or d appear in the output, those are the patterns appearing (i.e., they're treated as if they're sections in the output).

Also don't confuse the /d/ pattern with the command d. The command is always at the end in these demonstrations. The pattern is always between the //.

sed -n -e '/b/,/d/!p' abcde => ae
sed -n -e '/b/,/d/p' abcde => bcd
sed -n -e '/b/,/d/{//!p}' abcde => c
sed -n -e '/b/,/d/{//p}' abcde => bd
sed -e '/b/,/d/!d' abcde => bcd
sed -e '/b/,/d/d' abcde => ae
sed -e '/b/,/d/{//!d}' abcde => abde
sed -e '/b/,/d/{//d}' abcde => ace

SLePort · Answer 4 · 2017-06-30T05:04:50.777

19

Another approach with sed:

sed '/^#/,/^\$/{//!d;};' file

/^#/,/^\$/: from line starting with # up to next line starting with $
//!d: delete all lines except those matching the address patterns

edited Jun 30 '17 at 05:04

answered Jun 29 '17 at 19:21

SLePort

14,775
3
30
41

1

how do you do this inclusive of the patterns? – qodeninja Dec 30 '17 at 01:53
6

Try this: `sed '/^#/,/^\$/d;' file`. – SLePort Dec 30 '17 at 05:28
How to delete including only the first or only the last pattern? – t7e Jun 03 '22 at 15:35

score 4 · Answer 5 · answered Jun 09 '11 at 03:34

I did something like this long time ago and it was something like:

sed -n -e "1,/# ID 1/ p" -e "/\$ description 1/,$ p"

Which is something like:

-n suppress all output
-e "1,/# ID 1/ p" execute from the first line until your pattern and p (print)
-e "/\$ description 1/,$ p" execute from the second pattern until the end and p (print).

I might be wrong with some of the escaping on the strings, so please double check.

score 0 · Answer 6 · answered Jan 30 '20 at 17:09

The example below removes lines between "if" and "end if".

All files are scanned, and lines between the two matching patterns are removed ( including them ).

IFS='
'
PATTERN_1="^if"
PATTERN_2="end if"

# Search for the 1st pattern in all files under the current directory.
GREP_RESULTS=(`grep -nRi "$PATTERN_1" .`)

# Go through each result
for line in "${GREP_RESULTS[@]}"; do

   # Save the file and line number where the match was found.
   FILE=${line%%:*}
   START_LINE=`echo "$line" | cut -f2 -d:`

   # Search on the same file for a match of the 2nd pattern. The search 
   # starts from the line where the 1st pattern was matched.
   GREP_RESULT=(`tail -n +${START_LINE} $FILE | grep -in "$PATTERN_2" | head -n1`)
   END_LINE="$(( $START_LINE + `echo "$GREP_RESULT" | cut -f1 -d:` - 1 ))"

   # Remove lines between first and second match from file
   sed -e "${START_LINE},${END_LINE}d;" $FILE > $FILE

done

Using sed to delete all lines between two matching patterns

6 Answers6

OUTPUT

Explanation:

Linked