2

I'm new to the Salesforce community, so I apologize if this question is asked wrong.

I was running into essentially the exact same problem mentioned in this post. Basically, I have a query that exceeds 2000 records. The answerer says I should use the function queryMore.

However, one of my co-workers suggested that I could get around this by using the BULK api. The BULK Api seems to work and I'm able to successfully retrieve all 32000 records in our database. I guess I'm wondering if there is a reason to be cautious with the BULK api. It seems the accepted solution to my problem is to use queryMore.

My Use Case

I am creating a RESTful service that returns the Salesforce data on-demand. I would store the Salesforce data locally, and I would update on some regular interval. Written using Node.js with jsforce, this is what it might look like. 

function update_progress() {
  jsforce-conn.query(progress_query, function (err, result) {
    if (err) {
      return console.error(err);
    }
    apidata.progress = result;
  });
}

// http://localhost:4001/progress
express-app.get('/progress', (req, res) => { 
  res.setHeader('Content-Type', 'application/json');
  res.send(apidata.sf.progress.records);
});

Note that I'm not doing anything with files. Rather I'm storing all the queried information inside of a node server.

My question

Should I use Bulk? Or would it be better to use SOQL queries with a for-loop or queryMore?

Cameron
  • 147
  • 7
  • 2
    What are you actually trying to do with your query - are you pulling data out in bulk and asynchronously, doing a one-time extraction, rendering information for a user to view, something else? In what language or context are you working? – David Reed Feb 07 '19 at 17:37
  • I am creating a RESTful service that serves up this data and updates it on regular intervals (i.e. maybe 5 or 10 minutes). – Cameron Feb 07 '19 at 17:53
  • I've updated my question – Cameron Feb 07 '19 at 17:56

1 Answers1

3

Synchronizing the entire data set every 5-10 minutes is very inefficient, whether you are using the Bulk or REST API.

While the Bulk API could work for this purpose, it is an asynchronous API, and you don't have any performance guarantees as to when your Contact extraction will complete. Additionally, it comes with its own unique limits, and while your proposed use case sounds like it would not hit those limits, its possible that future expansion of your application would need to be concerned about the in concert with everything else your org's doing.

There are other APIs that are specifically designed to support replication and provide you with the ability to extract changed or updated records at intervals. You should at least consider using the getUpdated() and getDeleted() SOAP API calls for your regular updates, coupled with the Bulk API for an initial load of your entire database.

Change Data Capture is another feature to be aware of in supporting data replication, although I think you'd probably have to do more work to use it for this application.

David Reed
  • 92,733
  • 13
  • 84
  • 157
  • Thanks, this is a better answer than what I was expecting. I think you're right that this would be a better design. – Cameron Feb 07 '19 at 18:55