6

I have asked this question in biostars, but I am trying here as well as people working with "omics" data might be able to help. I think my issue relates understanding how large data storage on online server works. I am trying to download large data from the Human Microbiome Project. These are available on online server and can be downloaded via aspera (ascp).

I can't figure out the right path to get the data. On their website, they give an example of command to retreive an example dataset:

 ascp -l 100M -QT user@aspera.hmpdacc.org:/ReferenceGenomes/test_genome.tar /path/to/local/directory

I executed this, using my own username, from this I am asked my password, and I can authenticate. But then I get the following error:

(Error: Server aborted session: No such file or directory)

Then, I downloaded the manifest file via the HMP portal. In this manifest, they provide the path to which these files are supposed to be available. I tried to execute again the command with one of those path, provided by the manifest file:

ascp -l 100M -QT user@aspera.hmpdacc.org:/files/5a950f27980b5d93e4c16da124000378  /path/to/local/directory

I again get the same error of "No such file or directory".

The full url provided by the manifest

I tried putting different chunks of this address into to ascp command but nothing works. I think there is something I don't understand about how to access the server via aspera and what is expected to go in this path. When I copy paste this url in google, this directly download the file I want. So I'm close, but I need to do it via command line as I need to download hundreds of these files and they are huge (6-10GB each).

How can I download the data of the human microbiome project using ascp?

llrs
  • 4,693
  • 1
  • 18
  • 42
CAsimonet
  • 61
  • 1
  • Is the manifest file a list of links like you provided? I mean to ask is the manifest file hundreds of lines with a URLs like http://downloads.hmpdacc.org/data2/hhs/genome/microbiome/wgs/analysis/hmwgsqc/v2/SRS013951.tar.bz2per line? It would helpful to see the format of the manifest file. The first 5 lines should suffice. – Kohl Kinning Jul 04 '18 at 19:15

1 Answers1

-1

I can't test for ascp because they do not longer support my current setup but you can follow the FTP instructions. With FileZilla:

url: public-ftp.ihmpdcc.org

llrs
  • 4,693
  • 1
  • 18
  • 42
  • It looks like @CAsimonet has already gone through the instructions on the website and is have fine-tuned issues with getting a lot of specific files. – Kohl Kinning Jul 04 '18 at 19:59