1

I need to save files I get from S3 into a Lambda's file system and I wanted to know if I can do that simply using fs.writeFileSync ?

Or do I have to still use the context function as described here: How to Write and Read files to Lambda-AWS with Node.js

(tried to find newer examples, but could not).

What is the recommended method?

Please advise.

David Faizulaev
  • 3,385
  • 15
  • 54
  • 92
  • 1
    Not sure what you mean by the "context function". Yes, you can use the typical `fs` functions, but writing is limited to /tmp and the max diskspace available to your Lambda function in that location is 512 MB. Files written there may persist to the next (warm) Lambda invocation. – jarmod Oct 07 '21 at 16:34
  • If simply copying from S3 to local disk then stream the content ([example](https://stackoverflow.com/a/33938065/271415)). – jarmod Oct 07 '21 at 16:40
  • If I steam the content, what diskspace limit do I have? – David Faizulaev Oct 07 '21 at 17:38
  • You can write up to 512MB in /tmp. Depending on your app, you may be able to process the entire S3 object in RAM via streaming, without any need to persist to disk. – jarmod Oct 07 '21 at 17:43
  • please post your comment as an answer so I will accept it. – David Faizulaev Oct 10 '21 at 13:16

2 Answers2

2

Yes, you can use the typical fs functions to read/write from local disk, but be aware that writing is limited to the /tmp directory and the max diskspace available to your Lambda function in that location is 512 MB. Also note that files written there may persist to the next (warm) Lambda invocation.

If you want to simply download an object from S3 to the local disk (assuming it will fit in the available diskspace) then you can combine AWS SDK methods and Node.js streaming to stream the content to disk.

Also, it's worth noting that, depending on your app, you may be able to process the entire S3 object in RAM via streaming, without any need to actually persist to disk. This is helpful if your object size is over 512MB.

If you need to persist very large files, consider using Elastic File System.

jarmod
  • 59,580
  • 13
  • 95
  • 104
0

Lambda does not allow access to the local file system. It is mean to be an ephemeral environment. They allow access to the /tmp folder, but only for a maximum of 512MB. If you want to have storage along with your function, you will need to implement AWS S3 or AWS EFS.

Here's an article from AWS explaining this.

Here's the docs on adding storage to Lambda.

zbauman
  • 152
  • 7
  • 1
    AWS Lambda *does* allow access to the local file system. The effective (sandbox) user may not necessarily have sufficient permission to access the entire file system but it can certainly access all-readable files e.g. in /etc. – jarmod Oct 07 '21 at 17:41
  • 1
    @jarmod that’s a good distinction, thanks – zbauman Oct 07 '21 at 18:12
  • Not sure I understand, I can direct a lambda to use a S3 bucket as its storage? – David Faizulaev Oct 07 '21 at 18:31