Extract text between two strings repeatedly using sed or awk?

Question

I have a file called 'plainlinks' that looks like this:

13080. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94092-2012.gz
13081. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94094-2012.gz
13082. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94096-2012.gz
13083. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94097-2012.gz
13084. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94098-2012.gz
13085. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94644-2012.gz
13086. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94645-2012.gz
13087. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94995-2012.gz
13088. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94996-2012.gz
13089. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-96404-2012.gz

I need to produce output that looks like this:

999999-94092
999999-94094
999999-94096
999999-94097
999999-94098
999999-94644
999999-94645
999999-94995
999999-94996
999999-96404

Chris Seymour · Accepted Answer · 2012-11-14T21:18:08.920

Using sed:

sed -E 's/.*\/(.*)-.*/\1/' plainlinks

Output:

999999-94092
999999-94094
999999-94096
999999-94097
999999-94098
999999-94644
999999-94645
999999-94995
999999-94996
999999-96404

To save the changes to the file use the -i option:

sed -Ei 's/.*\/(.*)-.*/\1/' plainlinks

Or to save to a new file then redirect:

sed -E 's/.*\/(.*)-.*/\1/' plainlinks > newfile.txt

Explanation:

s/    # subsitution
.*    # match anything
\/    # upto the last forward-slash (escaped to not confused a sed)
(.*)  # anything after the last forward-slash (captured in brackets)
-     # upto a hypen
.*    # anything else left on line
/     # end match; start replace 
\1    # the value captured in the first (only) set of brackets
/     # end

Thanks a ton that did it – Mike Furlender Nov 14 '12 at 20:02 — Mike Furlender, Nov 14 '12 at 20:02

score 7 · Answer 2 · answered Nov 14 '12 at 20:02

7

Just for fun.

awk -F\/ '{print substr($7,0,12)}' plainlinks

or with grep

grep -Eo '[0-9]{6}-[0-9]{5}' plainlinks

answered Nov 14 '12 at 20:02

matchew

18,350
4
42
46

2

+1 for simpler grep solution. – Chris Seymour Nov 14 '12 at 20:11
@sudo_o well thanks, +1 for your solution. for being first. – matchew Nov 14 '12 at 20:25
agreed, +1 for the elegant grep solution – sampson-chen Nov 14 '12 at 20:42
@sampson-chen, ok, +1 for you too. – matchew Nov 14 '12 at 20:47

sampson-chen · Answer 3 · 2012-11-14T19:52:11.163

Assuming the format stays consistent as you have described, you can do it with awk:

awk 'BEGIN{FS="[/-]"; OFS="-"} {print $7, $8}' plainlinks > output_file

Output:

999999-94092
999999-94094
999999-94096
999999-94097
999999-94098
999999-94644
999999-94645
999999-94995
999999-94996
999999-96404

Explanation:

awk reads your input file one line at a time, breaking each line into "fields"
'BEGIN{FS="[/-]"; OFS="-"} specifies that delimiter used on the input lines should be either / or -, it also specifies that the output should be delimited by -
{print $7, $8}' tells awk to print the 7th and 8th field of each line, in this case 999999 and 9xxxx
plainlinks is the where the name of the input file would go
> output_file redirects output to a file named output_file

score 4 · Answer 4 · answered Nov 14 '12 at 19:54

4

Just with the shell's parameter expansion:

while IFS= read -r line; do
    tmp=${line##*noaa/}
    echo ${tmp%-????.gz}
done < plainlinks

answered Nov 14 '12 at 19:54

glenn jackman

223,850
36
205
328

score 1 · Answer 5 · answered Nov 15 '12 at 01:36

1

If the format stays the same, no need for sed or awk:

cat your_file | cut -d "/" -f 7- | cut -d "-" -f 1,2

answered Nov 15 '12 at 01:36

jfg956

15,207
3
24
34

If the format doesn't stay the same, the sed and awk solutions will break just as much as this. :) – Kaz Oct 08 '13 at 00:42

Extract text between two strings repeatedly using sed or awk?

5 Answers5

Linked