Get Recursive folder using SFTP and Paramiko Python

Question

Trying to implement a python script to and put get recursive folder from remote machine to local machine , this code works for copy : put folders and get folders , now i want to create move , the idea is to remove file once transfered i add line sftp.remove() and get the error stack at the end of the post any clue guys ?

import paramiko
    import os
    from stat import S_ISDIR, S_ISREG  
    
    class MySFTPClient(paramiko.SFTPClient):
            def put_dir(self, source, target):
                    ''' Uploads the contents of the source directory to the target path. The
                        target directory needs to exists. All subdirectories in source are 
                        created under target.
                    '''
                    for item in os.listdir(source):
                        if os.path.isfile(os.path.join(source, item)):
                            self.put(os.path.join(source, item), '%s/%s' % (target, item))
                        else:
                            self.mkdir('%s/%s' % (target, item), ignore_existing=True)
                            self.put_dir(os.path.join(source, item), '%s/%s' % (target, item))
    
            def mkdir(self, path, mode=511, ignore_existing=False):
                    ''' Augments mkdir by adding an option to not fail if the folder exists  '''
                    try:
                        super(MySFTPClient, self).mkdir(path, mode)
                    except IOError:
                        if ignore_existing:
                            pass
                        else:
                            raise
    
    def sftp_get_recursive(path, dest, sftp):
            item_list = sftp.listdir_attr(path)
            dest = str(dest)
            if not os.path.isdir(dest):
                os.makedirs(dest, exist_ok=True)
            for item in item_list:
                mode = item.st_mode
                if S_ISDIR(mode):
                    sftp_get_recursive(path + "/" + item.filename, dest + "/" + item.filename, sftp)
                else:
                    sftp.get(path + "/" + item.filename, dest + "/" + item.filename)
                    print("Removing file :",item.filename)
                    sftp.remove(dest + "/" + item.filename)

Solved by changing sftp.remove(dest + "/" + item.filename) to sftp.remove(path + "/" + item.filename)

    transport = paramiko.Transport(('172.31.11.233', 22))
    transport.connect(username='$(ops_unv_cred_user_018f3e16e789465bb5110a90837ce03f)', password='$(ops_unv_cred_pwd_018f3e16e789465bb5110a90837ce03f)')
      
    if 1==1:
        sftp = paramiko.SFTPClient.from_transport(transport)            
        if 0==1:
            sftp.get('/home/karim/EDV','/home/karim/israel')
            print("copied successfully!")                      
        else:
            sftp_get_recursive('/home/karim/EDV', '/home/karim/israel', sftp)
            print("copied successfully!")
            if 1==1:
                print("Listing all files in remote machine where path is => /home/karim/EDV :" )
                print(sftp.listdir('/home/karim/EDV'))
        if 1==1: 
            print("Listing all files in remote machine where path is => /home/karim/EDV :" )
            print(sftp.listdir('/home/karim/EDV'))       
    else:
        sftp = MySFTPClient.from_transport(transport) 
        if 0==1: 
            sftp.put('/home/karim/israel','/home/karim/EDV')
            print("copied successfully!")
           
        else:        
            sftp.mkdir('/home/karim/EDV', ignore_existing=True)
            sftp.put_dir('/home/karim/israel','/home/karim/EDV')
            print("copied successfully!")
    
        if 1==1:   
            print("Listing all files in remote machine where path is => /home/karim/EDV :" )
            print(sftp.listdir('/home/karim/EDV'))
    
    sftp.close()



Traceback (most recent call last):   File "/home/karim/.56ee3cce-0616-4070-bfa5-b5a0f346b2c7.sh", line 64, in <module>
    sftp_get_recursive('/home/karim/EDV', '/home/karim/israel', sftp)   File "/home/karim/.56ee3cce-0616-4070-bfa5-b5a0f346b2c7.sh", line 50, in sftp_get_recursive
    sftp_get_recursive(path + "/" + item.filename, dest + "/" + item.filename, sftp)   File "/home/karim/.56ee3cce-0616-4070-bfa5-b5a0f346b2c7.sh", line 54, in sftp_get_recursive
    sftp.remove(dest + "/" + item.filename)   File "/opt/universal/python/lib/python3.7/site-packages/paramiko/sftp_client.py", line 398, in remove
    self._request(CMD_REMOVE, path)   File "/opt/universal/python/lib/python3.7/site-packages/paramiko/sftp_client.py", line 813, in _request
    return self._read_response(num)   File "/opt/universal/python/lib/python3.7/site-packages/paramiko/sftp_client.py", line 865, in _read_response
    self._convert_status(msg)   File "/opt/universal/python/lib/python3.7/site-packages/paramiko/sftp_client.py", line 894, in _convert_status
    raise IOError(errno.ENOENT, text) FileNotFoundError: [Errno 2] No such file

What exactly is your question? `No such file or directory: '/home/karim/EDV'` what is unclear about this? — mkrieger1, Dec 27 '21 at 15:19
You are right , sorry i used Import instead ,just error while posting. so the error is with import Paramiko — Karim Azzouzi, Dec 27 '21 at 15:21
The path /home/karim/EDV exist in remote machine and include some files and folders. seems it does not walk trough os.listdir(target) — Karim Azzouzi, Dec 27 '21 at 15:25

Martin Prikryl · Accepted Answer · 2021-12-27T15:47:48.720

0

From your comments, it seems that the target is a remote path, although it should be "source", as you are downloading. And you are using the remote path with local API like os.listdir. That cannot work, you should use SFTPClient.listdir_attr.

For an example of working recursive download, see:
Recursive directory download with Paramiko?

edited Dec 27 '21 at 15:47

answered Dec 27 '21 at 15:33

Martin Prikryl

167,268
50
405
846

The problem with pysftp , is that Downloading recursive folder from windows to linux does not work isnt it ? – Karim Azzouzi Dec 27 '21 at 15:40
I do not understand your comment. You are not downloading from Windows to Linux, right? (and pysftp has problem with download from Linux to Windows, not form Windows to Linux, anyway). And I'm not suggesting you to use pysftp. The link I have posted contains code that with tiny modification (described there) works with Paramiko too. Please follow the link. – Martin Prikryl Dec 27 '21 at 15:44
Or see https://stackoverflow.com/a/53556194/850848#53556194 – Martin Prikryl Dec 27 '21 at 15:47
Thanks for the clue , but how do you use the get_r_portable like this ? transport = paramiko.Transport(('${ops_sftp_hostname}', 22)) transport.connect(username='${_credentialUser('${ops_sftp_credential}')}', password='${_credentialPwd('${ops_sftp_credential}')}') sftp = paramiko.SFTPClient.from_transport(transport) sftp.get_r_portable('${ops_sftp_remote_path}','${ops_sftp_local_path}') sftp.close() – Karim Azzouzi Dec 27 '21 at 16:47
I got : Traceback (most recent call last): File "/home/karim/.f041aed9-fe22-4371-b350-cbecbb5ce567.sh", line 21, in get_r_portable(sftp,'/home/karim/EDV','/home/karim/nouveau') File "/home/karim/.f041aed9-fe22-4371-b350-cbecbb5ce567.sh", line 15, in get_r_portable get_r_portable(sftp, remotepath, localpath, preserve_mtime) NameError: name 'preserve_mtime' is not defined – Karim Azzouzi Dec 27 '21 at 16:53
Ok, I've corrected the code. The `preserve_mtime` should not be there. And you should us the function like `get_r_portable(sftp, '${ops_sftp_remote_path}','${ops_sftp_local_path}')` – Martin Prikryl Dec 27 '21 at 20:11
Thanks Man , so i'm currently stucking while i want to remove a file with paramiko : -------------------------------- for item in item_list: mode = item.st_mode if S_ISDIR(mode): sftp_get_recursive(path + "/" + item.filename, dest + "/" + item.filename, sftp) else: sftp.get(path + "/" + item.filename, dest + "/" + item.filename) print("Removing file :",item.filename) sftp.remove(dest + "/" + item.filename) ------------------------------- – Karim Azzouzi Jan 05 '22 at 14:37
I think i did understand my mistake , when we place sftp.remove(dest+"\"+item.filename) the destination here point to a static path, however the item.filename can be in different subfolders. so i should find a way to get each time the right path. – Karim Azzouzi Jan 05 '22 at 15:19
If your point is to delete the *remote* file that you have just downloaded, then use the same path as in the first argument of `sftp.get(remotepath, localpath)`. So if you are doing `sftp.get(path + "/" + item.filename, ...`, then do `sftp.remove(path + "/" + item.filename)`. – Martin Prikryl Jan 05 '22 at 15:48
sftp.remove(path + "/" + item.filename) this is exactly what drive me at the end of the initial post , i don't know what's wrong with that ? – Karim Azzouzi Jan 05 '22 at 15:50
Solved , you are genius , i was pointing on dest folder rather then path, my bad. – Karim Azzouzi Jan 05 '22 at 15:53
guys , how can i check if all folders are empty return exit code 1 this code does not work for me : for item in item_list: if not str(item) : print(item.filename) print("No file found") sys.exit(1) break – Karim Azzouzi Jan 06 '22 at 14:57
Please [accept this answer](https://stackoverflow.com/help/someone-answers), before asking new questions. – Martin Prikryl Jan 06 '22 at 15:01
How ot accepte i have notghin to check to accept it ? – Karim Azzouzi Jan 06 '22 at 16:20
Did you follow the link? – *"To mark an answer as accepted, click on the check mark beside the answer to toggle it from greyed out to filled in."* – Martin Prikryl Jan 06 '22 at 17:00

Get Recursive folder using SFTP and Paramiko Python

1 Answers1