1

I am trying to convert my CSV file into JSON by uploading it into FastAPI first, but when I try to process it directly (without storing it somewhere), I get error:

Error : FileNotFoundError: [Error 2] No such file or directory : "testdata.csv"

Code :

async def upload(file: UploadFile = File(...)):
    data = {}    
    with open(file.filename,encoding='utf-8') as csvf:
        csvReader = csv.DictReader(csvf)
        for rows in csvReader:             
            key = rows['No']
            data[key] = rows    
    return {data}```

Chris
  • 4,940
  • 2
  • 7
  • 28
sourabh
  • 11
  • 3
  • What is the output of `os.getcwd()`, and is it the same as the location of `testdata.csv`? – BrokenBenchmark Jan 07 '22 at 06:00
  • Actually I am Directly uploading file in UI and I am not storing anywhere so when I use is getcwd() command I do get 200 response code but response body [ null ] – sourabh Jan 07 '22 at 06:22

3 Answers3

1

UploadFile uses Python's SpooledTemporaryFile, a "file stored in memory", and which "is destroyed as soon as it is closed". For more info on that, please have a look at this answer.

Option 1

To approach the problem on your way (i.e., reading from csv file and not using the file contents that you can get from contents = await file.read()), you can copy the file contents into a NamedTemporaryFile (again, check this answer out for more info on that), and then use it to iterate over the csv contents. Below is a working example:

import uvicorn
from fastapi import FastAPI, File, UploadFile
from tempfile import NamedTemporaryFile
import os
import csv

app = FastAPI()
    

@app.post("/upload")
async def upload(file: UploadFile = File(...)):
    contents = await file.read()
    data = {}
    file_copy = NamedTemporaryFile(delete=False)
    
    try:
        with file_copy as f:  # The 'with' block ensures that the file closes and data are stored
            f.write(contents);
        
        with open(file_copy.name,'r', encoding='utf-8') as csvf:
            csvReader = csv.DictReader(csvf)
            for rows in csvReader:             
                key = rows['No']
                data[key] = rows  
    finally:
        file_copy.close()  # Remember to close any file instances before removing the temp file
        os.unlink(file_copy.name)  # delete the file
    
    return data
    

Option 2

Alternatively, a much more elegant solution would be to use, as mentioned earlier, the byte data of the uploaded file, saving you from copying them into a new temporary file. Convert the bytes into a string, and then load the string object into an in-memory text buffer (i.e., StringIO), as mentioned here, which you can pass to the csv reader. Example below:

from fastapi import FastAPI, File, UploadFile
import csv
from io import StringIO

app = FastAPI()

@app.post("/upload")
async def upload(file: UploadFile = File(...)):
    data = {}
    contents = await file.read()
    decoded = contents.decode()
    buffer = StringIO(decoded)
    csvReader = csv.DictReader(buffer)
    for rows in csvReader:             
        key = rows['No']
        data[key] = rows  
        
    buffer.close()
    return data

Option 3

You could also write the bytes from the uploaded file to a BytesIO stream, which you could then convert into a pandas dataframe. Next, using the to_dict() method, you could convert the dataframe into dictionary and return it (which, by default, FastAPI will convert into JSON using the jsonable_encoder and return a JSONResponse).

from fastapi import FastAPI, File, UploadFile
from io import BytesIO
import pandas as pd

app = FastAPI()
    
@app.post("/upload")
async def upload(file: UploadFile = File(...)):
    contents = await file.read()
    buffer = BytesIO(contents)
    df = pd.read_csv(buffer)
    buffer.close()
    return df.to_dict(orient='records')
Chris
  • 4,940
  • 2
  • 7
  • 28
  • The `file.read()` must be `file.file.read()` – snowmanstark Mar 22 '22 at 13:35
  • @snowmanstark All `UploadFile` methods "_call the corresponding file methods underneath (using the internal `SpooledTemporaryFile`)_". Please have a look at the [documentation](https://fastapi.tiangolo.com/tutorial/request-files/#uploadfile) – Chris Mar 22 '22 at 13:53
  • I was wrong in saying that "The `file.read()` must be `file.file.read()` ". If we want to convert """ contents = await file.read() decoded = contents.decode() buffer = StringIO(decoded) """ then it would be `buffer = StringIO(file.file.read().decode())` – snowmanstark Mar 22 '22 at 14:19
  • You **shouldn't** change _asynchronous_ reading of the contents to _synchrnonous_, unless you have decided to declare your route with `def`, not `async def`. Doing so (while the route is declared with `async def`) would result in blocking the entire server, until that operation is completed. Please have a look at the following references to understand the concept of `async/await`: [1](https://stackoverflow.com/a/71517830/17865804), [2](https://stackoverflow.com/a/67601373/17865804), [3](https://fastapi.tiangolo.com/async/). – Chris Mar 22 '22 at 16:04
  • 1
    Thanks Chris, I get your point! – snowmanstark Mar 22 '22 at 16:44
0

The reason why you are getting the Error : FileNotFoundError: [Error 2] No such file or directory : "testdata.csv" is because you are trying to read a file that is not stored locally.

If you want to read the file this way you should save the uploaded file before proceeding:

async def upload(uploaded_file: UploadFile = File(...)):
    # save csv to local dir
    csv_name = uploaded_file.filename
    csv_path = 'path_to/csv_dir/'
    file_path = os.path.join(csv_path, csv_name)
    with open(file_path, mode='wb+') as f:
        f.write(uploaded_file.file.read())

    # read csv and convert to json
    data = {}
    with open(file_path, mode='r', encoding='utf-8') as csvf:
        csvReader = csv.DictReader(csvf)
        for rows in csvReader:             
            key = rows['No']
            data[key] = rows    
    return {data}
Lars
  • 89
  • 1
  • 6
  • But I was trying to do it directly like when file is uploaded in UI directly it should have been fetched and converted to JSON format and later I'll save that JSOn data to MySQL database but my code is fetching just name of uploaded file and not the actual file itself – sourabh Jan 08 '22 at 15:08
  • Have you tried `content = await file.read()` directly after the function definition? After that you can use the `file.content_type` property to get the file extension to determine the processing method. – Lars Jan 10 '22 at 07:10
0

The file in the async function upload() is already open and you can fetch characters from it directly, there's no need to open it again. Also in FastAPI the class UploadFile is actually derived from standard library tempfile.SpooledTemporaryFile, which cannot be accessed by specifying path of the temporary file.

For example , if you use CPython and read the value of file.filename in the upload() in the Unix-like system, it returns a number instead of a well-formed path, because any instance of the class SpooledTemporaryFile will create a file descriptor (at some point when current stored data exceeds max_size) and simply return the file descriptor (should be a number in Unix) on accessing SpooledTemporaryFile.filename

Han
  • 485
  • 6
  • 13