Using bulk API, do seperate serial batches still run parallel to eachother?

Question

I have to load several million child records distributed over about 2000 parent records, so obviously I'm running into locking issues. I'm trying to find a way to avoid locks as much as possible, so I came up with this:

I split my child record file into one file for every parent record. then I use Dataloader CLI to start a separate serial bulk API job for each of these files. My idea was that each batch job would run independently from the others while the individual batches for each job would not interfere with eachother.

However, from small scale testing it looks like the jobs don't run concurrently. A batch from Job A runs alone and only when it is done does a batch from Job B start running, and so on.

Is this expected behavior or is it just an anomaly? If several serial jobs are running at the same time, will they still finish in the same time as one large serial job?

I tried this, but the amount of children per parent varies too much. I can't seem to find a magic number to stop locks from happening. — Valyrion, Jul 23 '18 at 10:11
Have you read The Salesforce Bulk API - Maximizing Parallelism and Throughput Performance When Integrating or Loading Large Data Volumes? Some good tips and approaches to consider in there. — Daniel Ballinger, Jul 23 '18 at 22:56

score 1 · Accepted Answer · answered Jul 23 '18 at 11:25

Whether they will run parallel or not is something that will be beyond your control. The general answer is that it is not likely for that to happen. The reason for that is because of how queued asynchronous jobs are processed in a pod. Rather than repeat that explanation here, I'm going to refer you to my answer to Dramatic increase in AsyncApexJob latency which explains how queued jobs are processed in a pod. The summary being that more jobs you have queued, the larger delays you can expect to occur between processing of your jobs.

score 1 · Answer 2 · answered Jul 24 '18 at 06:41

Using bulk API, do separate serial batches still run parallel to each other?

I have monitored jobs during my bulk data loading and if you define batch size is 2000 and in your csv there are 100k records then there are 50 jobs will get created.

Now, Salesforce first creates those files at their server and try to process those files. That's why initially for this file creation you could expect some delay in data loading.

If you define serial mode, then files will be processed serially one by one and when one job is getting processed, the other job will be in queue.

Secondly, during the processing for each job, bulk API process the dataset in terms of 200 records per batch in that file.

Is this expected behavior or is it just an anomaly?

Yes, it is expected behavior and as per design.

If several serial jobs are running at the same time, will they still finish in the same time as one large serial job?

If mode is serial then files are processed serially and maximum batch size for serial mode can be defined as 2000. Based on the batch size defined, the no. of files created (total rows in csv/batch size). So, lesser the batch size, more are the files. So, total processing time will be higher because of file creation.

So, several jobs are not running same time for serial mode. Those jobs run in parallel during parallel mode.

You can monitor jobs navigating this path:

Setup->Monitoring-> Bulk Data Load Jobs

Using bulk API, do seperate serial batches still run parallel to eachother?

2 Answers2