2

I have multiple directories with the same file (remote) and I want to copy them to multiple directories on my local machine.

So lets say remote I have

~/remote_folder/folder1/Myfile.log
~/remote_folder/folder2/Myfile.log
~/remote_folder/folder3/Myfile.log

Do note that next to Myfile.log there are also other files in the folder directories.

And local I want:

~/local_folder/folder1/Myfile.log
~/local_folder/folder2/Myfile.log
~/local_folder/folder3/Myfile.log

Here I only want the Myfile.log and not the other files from the remote.

So what I first had was (I use a bash script):

for folder in ~/remote_folder/*/
do
                dirr="$(basename $folder)" 
                scp "$folder"Myfile.log user@host:~/local_folder/"$dirr"/
done

However this only works for the first 5 folders and then I get connection refused, probably because to many ssh connections.

Does there exist something that I could do this

scp ~/remote_folder/*/Myfile.log user@host:~/local_folder/*/

or something similar, but at least that I can do it with only one ssh connection?

C. Binair
  • 123
  • Wildcards should work – Raman Sailopal Oct 15 '20 at 15:02
  • As far as I understand wildcards only work in the source and not in the destination. –  Oct 15 '20 at 15:05
  • I understand folder1/, folder2/ etc. may not exist in local_folder/ and we need to create them, according to what's in the remote_folder/. Please confirm. Or do all the directories exits already? Can remote_folder/ contain folderN/ where there is no Myfile.log? Anyway it may be easier if you could treat all paths as local, like with SSHFS. Can you? Please clarify by [edit]ing the question. – Kamil Maciorowski Oct 15 '20 at 15:41
  • You can assume all the directories exist already if that makes it easier. And assume remote_folder/ contains only folderN's with Myfile.log. What do you mean it may be easier if you could treat all paths as local? – C. Binair Oct 15 '20 at 16:04
  • @C.Binair SSHFS allows you to work with remote files as if they were local. Then you can use tools like find or cp that know nothing about SSH. In this case the most important thing may be that wildcards in a local shell will be able to match local paths that really lead to remote files. If you tell me you can use SSHFS then I will probably be able to help you. Can you install SSHFS on the local computer? Or can you have it installed? – Kamil Maciorowski Oct 15 '20 at 17:28

1 Answers1

1

You can use rsync for this:

rsync -r --prune-empty-dirs --include='/*/' --include='/*/Myfile.log' --exclude='*' user@host:remote_folder/ local_folder/

The include/exclude patterns are the only slightly tricky part here. Rsync includes or excludes based on the first match. And only directories that are included by patterns are recursed into. So we use:

  1. -r to do a recursive copy.
  2. --include='/*/' to recurse into each first-level sub-directory of the sync root.
  3. --include='/*/Myfile.log' to include the specific file that you want to sync out of each directory.
  4. --exclude='*' to ignore everything else.
  5. --prune-empty-dirs to prevent creating (empty) copies of directories where Myfile.log doesn't exist. You can leave this option off if every directory has a Myfile.log or if you don't care about empty directories getting created.

You can test it out by adding -n and -v options to rsync to see what would be copied before actually doing the copy.

Another option instead of rsync would be to use tar over an ssh pipe:

cd ~/local_folder
ssh user@host "cd ~/remote_folder && tar cf - */Myfile.log" | tar xf -

Personally, I prefer rsync because you can use -n -v to see exactly what you're going to get before doing any copying. But both solutions should give you the same result.

satwell
  • 356
  • Thank you! I was hoping that with my generalized situation I was able to distil the answer to my specific case but it seems I'm not capable. If my situation on the remote actually looks like ~/somefolder/myparentfolder/folder<n>/MyFile.log and local I want ~/Documents/somefolder1/somefolder2/myparentfolder/folder<n>/MyFile.log. How would the command look like then? EDIT: Oke never mind I found it. I just only need to change the source and target folders at the end of the command – C. Binair Oct 16 '20 at 07:45
  • I was hoping you could help me one more time. So now in my folder I have a folder (the same for every folder) and I have a file in that one that I want to copy. I tried (among other things) rsync -r --prune-empty-dirs --include='/*/' --include='/*/output/Myfile.log' --exclude='*' user@host:remote_folder/ local_folder/ but this doesn't find any file. I have tried multiple things but it doesn't seem to work and I don't see what I'm doing wrong. EDIT: nevermind I found it again. There needs to be another include after the first one --include='/*/output/' – C. Binair Feb 10 '21 at 09:44