Transferring files from FTP directly to Azure Blob storage in C#

Question

I have written a function which move files from FTP server to Azure Blob storage. I want to pass the stream from FTP to blob so that I can upload the files. I am running a while loop for every file and trying to move the file to blob storage using UploadFromStreamAsync(). But when I came to this call, my stream object gets disposed because of which file is getting transfer to blob but without any content. I do not want to dispose my stream object till all files are transfer. Can anyone tell me what's wrong going on??

public static async Task FTP_TO_BLOB_TRANSFER()
{
    string ftpPath = ConfigurationSettings.AppSettings.Get("ftpPath");
    string ftpUserName = ConfigurationSettings.AppSettings.Get("ftpUserName");
    string ftpPassword = ConfigurationSettings.AppSettings.Get("ftpPassword");

    FtpWebRequest request = (FtpWebRequest)WebRequest.Create(ftpPath);
    request.Method = WebRequestMethods.Ftp.ListDirectory;
    request.Credentials = new NetworkCredential(ftpUserName, ftpPassword);
    FtpWebResponse response = (FtpWebResponse)request.GetResponse();
    Stream responseStream = response.GetResponseStream();
    StreamReader reader = new StreamReader(responseStream);

    string connectionString = ConfigurationSettings.AppSettings.Get("connectionString");
    string folderName = "Inbox/";
    string file = reader.ReadLine();
    while (!string.IsNullOrEmpty(file))
    {

        string fileName = Path.GetFileNameWithoutExtension(file);
        string guid = Guid.NewGuid().ToString();
        string extension = Path.GetExtension(file);
        try
        {
            Stream fileForBlobStorage = reader.BaseStream;
            if (CloudStorageAccount.TryParse(connectionString, out storageAccount))
            {
                CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
                CloudBlobContainer cloudBlobContainer = cloudBlobClient.GetContainerReference("falcon");
                BlobContainerPermissions permissions = new BlobContainerPermissions
                {
                    PublicAccess = BlobContainerPublicAccessType.Blob
                };
                await cloudBlobContainer.SetPermissionsAsync(permissions);

                CloudBlockBlob cloudBlockBlob = cloudBlobContainer.GetBlockBlobReference(folderName + fileName + "-" + '[' + guid + ']' + guid + extension.ToString());

                await cloudBlockBlob.UploadFromStreamAsync((Stream  )fileForBlobStorage);
            }
            else
            {
                Console.WriteLine("Connection string not defined.");
            }
        }
        catch (Exception e)
        {
            string message = e.Message;
            Console.WriteLine(message);
        }
        file = reader.ReadLine();
    }
}

Answer 1

You are requesting directory listing of a folder on FTP server. And with the listing you are, at the same time:

• Reading the listing line-by-line (file-by-file) – somehow trying to process individual lines/files.
• Yet you are trying to upload the listing (the same stream) to the blob.

That can never work. And moreover it makes no sense.

I assume that you actually want to upload the files, not the listing.

For that you need to start downloading the individual files from the FTP server in your loop:

FtpWebRequest fileRequest = (FtpWebRequest)WebRequest.Create(ftpPath + file);
fileRequest.Method = WebRequestMethods.Ftp.DownloadFile;
fileRequest.Credentials = new NetworkCredential(ftpUserName, ftpPassword);
FtpWebResponse fileResponse = (FtpWebResponse)fileRequest.GetResponse();
Stream fileStream = fileResponse.GetResponseStream();

await cloudBlockBlob.UploadFromStreamAsync(fileStream);