finding unique headers in a fasta file using linux command line

Question

I tried to use the following command

uniq -u reference.fasta >> reference_uniq.fasta

I'd like a count of the unique headers.

Answer 1

If you just want the number of unique headers, you can do this:

grep '>' reference.fasta | sort | uniq | wc -l

If you want a list of the unique headers, you can do this:

grep '>' reference.fasta | sort | uniq

If you want a histogram of how many times each header occurs, you can do this:

grep '>' reference.fasta | sort | uniq -c | awk '{printf("%s\t%s\n", $1, $2)}'

Answer 2

The uniq command expects sorted input. Interestingly, the sort command actually has a "unique" option, -u, which means uniq is not strictly needed. For the fastest processing, you can look for the '>' character at the start of lines with grep:

grep '^>' reference.fasta | sort -u > reference_headers_unique.fasta

For returning the number of unique lines, pipe through wc -l:

grep '^>' reference.fasta | sort -u | wc -l

For more information about regular expressions, see here.

Answer 3

If, for some reason, you need to keep the original order of the input file, so you only want the first occurrence of each header, you can do:

awk '/^>/ && !a[$0]++' reference.fasta > headers

Or, to get the number only:

awk '/^>/ && !a[$0]++{k++}; END{print k}' reference.fasta 

However, these only make sense if you need to keep the order of the original file. If not, use Gringer's grep approach which will be much faster (reference.fasta is hg19):

$ time awk '/^>/ && !a[$0]++{k++}; END{print k}' reference.fasta 
93

real    0m34.510s
user    0m31.861s
sys     0m2.634s

$ time grep '^>' reference.fasta | sort -u | wc -l
93

real    0m16.597s
user    0m0.820s
sys     0m3.725s