96

In linux, I can grep a string from a file using grep mySearchString myFile.txt. How can I only get the result which are unique?

hap497
  • 2,959
  • 7
  • 32
  • 24

2 Answers2

153

You can achieve this with the sort and uniq utilities.

example:

[john@awesome ~]$ echo -e "test\ntest\ntest\nanother test\ntest"
test
test
test
another test
test
[john@awesome ~]$ echo -e "test\ntest\ntest\nanother test\ntest" | sort | uniq
another test
test

depending on the data you may want to utilize some of the switches as well.

  • 11
    @John T - I would recommend to use sort before uniq in case the data are not ordered. Otherwise uniq won't completely work. – Studer Feb 21 '10 at 02:58
  • t now I can upvote ! You also helped me writing others scripts here ;) – Studer Feb 21 '10 at 03:17
  • 56
    Use sort -u instead of sort | uniq. It saves a process, reduces the total I/O, and reduces the total number of comparisons that have to be made. – Chris Johnsen Feb 22 '10 at 05:55
  • @ChrisJohnsen You should make that comment an answer as it's a better solution then the current given answer – Nico Van Belle May 24 '18 at 07:05
18

You can use:

grep -rohP "(mySearchString)" . | sort -u

-r: recursive

-o: only print matching part of the text

-h: don't print filenames

-P: Perl style regex (you may use -E instead depending on your case)

sort -u is better than sort | uniq, as @Chris Johnsen pointed out.

Pato
  • 281
  • sort -R flag would use a random "sort" that only groups unique elements without actually sorting. I haven't benchmarked it, but it could be faster. – Dominykas Mostauskis Apr 03 '20 at 12:00