request.setEndPoint('callout:AWS_S3/Test/Demo.csv');
request.setHeader('Content-Type', 'application/xml');
request.setMethod('GET');
Http http= new Http ();
HTTPResponse response= http.send(request);
system.debug('Debug Request 1>>>'+response);
system.debug('Debug Request >>>'+response.getBody());
return request;
There are 0.5 million (500000) records in Demo CSV File saved in AWS S# Bucket 'Test' How do resolve this issue
To download part of a file, use the Range header, such as:
// Get the first MB of data
req.setHeader('Range', 'bytes=0-1000000');
Read more about this in the AWS GetObject documentation.
You'll need to do this for each chunk. Keep in mind that payload responses are limited to heap size per transaction, meaning you can't get more than 6MB of data total in a single synchronous Apex transaction, no matter how you split it up. You'll need Batchable/Queueable to handle the entire file.
Note also that you'll need to discover the total file size before starting, or check each response to see if all the data has been retrieved, because specifying an invalid range will return an error code instead of the remaining data. The RFC and GetObject documentation from above should tell you everything you need to handle that part.
Also, as a side note, you're going to also have to build a proper CSV parser with partial resume support. Make sure you fully understand how CSV is meant to work by reading something like RFC4180. I've made no assumptions about your CSV; you may run in to other problems, for example, if you have really large records in your CSV (>1Mb or so). Doing this in Apex is rather complicated.
Without knowing anything else about your problem (this is an X-Y Problem), I'd suggest you research doing this any other way. Write a NodeJS script and run it on Heroku. Make a Visualforce page and handle the data client-side. Set up CSP for Lightning and build a component to do this client-side. Apex is great for small projects, but this sounds like the most painful way to get things done, especially if it's a one-off time script.
