Downloading all COI sequences from BOLD fails

Question

I have metabarcoding sequence data (COI) from bulk animal samples (including arthropoda, nematoda, annelida, mollusca) and I want to BLAST all of these sequences. I used following command to do this: blastn -remote -db nt -query COI_all.fasta -num_alignments 2 -out COI_blasted.txt. However this results in errors similar to this post: https://www.biostars.org/p/359971/ .

These errors probably appear due to the number of sequences in my file (around 700) and the remote connection is thus interrupted.

I found that a solution would be to use blastn with a local database and since the samples are so diverse, I would like to download ALL animal COI sequences from BOLD (or gen bank). It would not be a problem if non-animal (e.g. plant) sequences would also be included.

I think the BOLD database would be great to BLAST my sequences to. However, I'm currently struggling to find a good way to download all animal COI sequences from BOLD.

When entering COI-5P as search term on http://v4.boldsystems.org/index.php/Public_SearchTerms I receive error: Your search terms resulted in too many matching terms. Please try again with more specific search criteria.. I could likely download the sequences from all the phyla etc seperately and merge them, but I'd rather just download 1 file.

I also tried to use the API by running: wget http://v4.boldsystems.org/index.php/API_Public/sequence?marker=COI-5P. A download starts but around 3.7 MB download, it is stuck and the file I receive only contains ~5000 sequences.

UPDATE: I've contacted BOLD about the stalling behavior and this is their reply: "This issue is because of the large API request that retrieves millions of records, which our system does not handle. Please break up the search by smaller groups, such as classes."

Does anyone have a solution to download all COI sequences from BOLD in one file?

I could also download COI sequences from gen bank using the ftp://ftp.ncbi.nlm.nih.gov/blast/db/ URL, but I'm not sure which exact files I need. For 16S, 18S,.. it is obvious, but not for COI. Any suggestions?

Thanks for the help.

Answer 1

I couldn't help but looking a bit deeper into this. There is actually an R package that uses the BOLD API.

However, they also say that there will be timeouts when downloading large sets of data. Luckily, they suggest a workaround (downloading sequences separately for each taxonomic Class).

So you should be able to download the marker sequences using the suggested steps:

# install packages (if necessary)
install.packages("bold")
install.packages("taxize")
install.packages("seqinr")
# load packages
library(tidyverse)
library(bold)    # API interface to BOLD
library(taxize)  # for NCBI taxonomy lookup
library(seqinr)  # for FASTA output

# get class-level taxa within "Animalia" from NCBI taxonomy
taxa <- downstream("Animalia", db = "ncbi", downto = "class")
# check if taxa present in BOLD
checks <- bold_tax_name(taxa$Animalia$childtaxa_name)
taxa_bold <- checks[!is.na(checks$taxon),]$taxon

# download sequences from BOLD for each class-level taxon
sequences <- map(taxa_bold[1:3], bold_seq, marker = 'COI-5P') %>%
  flatten() %>%
  bind_rows()

# write sequences to file
write.fasta(
  sequences = as.list(sequences$sequence), 
  names = as.list(sequences$id), 
  nbchar = 80, 
  file.out = 'coi5p.fasta')

Note that I selected the first tree taxa (taxa_bold[1:3]) for testing, so that it runs quicker. If you want to get all taxa, just remove the [1:3].

One last importand caveat: In the help for bold_seq it says that

Notes from BOLD on the marker param: "All markers for a specimen matching the search stringwill be returned. ie. A record with COI-5P and ITS will return sequence data for both markers evenif only COI-5P was specified."You will likely end up with data with markers that you did not request - just be sure to filter thoseout as needed.

So you might want to double-check that the downloaded sequences are actually COI-5P. But if you are using them as a BLAST database, you might also be happy with COI-5P and ITS markers being present in the database (it affects the e-values, though).

Answer 2

A straightforward way of getting the sequences from GenBank is from their website using this URL: https://www.ncbi.nlm.nih.gov/nuccore?term=COI[Gene Name]

There you can click on "Send to" -> select "Gene Features" and "FASTA Nucleotide" -> Click "Create File".

There are >3 mio hits, so the download could take a while.