How to catch duplicate entries in text file in linux

Question

Text file:

I want to catch 1 1 as duplicated

Lev Levitsky · Accepted Answer · 2015-08-09T18:06:41.393

105

Your question is not quite clear, but you can filter out duplicate lines with uniq:

sort file.txt | uniq

or simply

sort -u file.txt

(thanks RobEarl)

You can also print only repeating lines with

sort file.txt | uniq -d

edited Aug 09 '15 at 18:06

answered Oct 05 '12 at 09:57

Lev Levitsky

2

`sort -u file.txt` would also filter out duplicates – RobEarl Oct 05 '12 at 20:45
1

actually that works for me `sort file.txt | uniq -d` Thank, Lev – Kalin Borisov Oct 08 '12 at 07:10
2

@KalinBorisov No problem. Keep in mind that you can [accept helpful answers](http://meta.stackexchange.com/a/5235/181223). – Lev Levitsky Oct 08 '12 at 08:10
I will keep in mind thank you, Lev. – Kalin Borisov Oct 08 '12 at 10:53

score 2 · Answer 2 · answered Oct 05 '12 at 11:39

2

One way using GNU awk:

awk 'array[$0]++' file.txt

Results:

1 1

answered Oct 05 '12 at 11:39

Steve

score 1 · Answer 3 · answered Jul 21 '14 at 19:13

1

You can use it easily:

sort -u file.txt

OR

awk '!x[$0]++' file.txt

answered Jul 21 '14 at 19:13

MLSC

3 Answers3