0

I have a list of tasks to complete in asyncio, I want to execute them in batches rather than all at once.

I have created a list of tasks as and a test as follows:

async def my_sleep_function(t):
    print('Waiting {} seconds'.format(t))
    await asyncio.sleep(t)

async def task_split_test():
    tasks = []
    for i in range(10):
        tasks.append(asyncio.create_task(my_sleep_function(i)))

    print("Starting full task list")
    t = time()
    await split_tasks(tasks, 10)
    print("Time Taken: {}".format(time() - t))

    tasks = []
    for i in range(10):
        tasks.append(asyncio.create_task(my_sleep_function(i)))

    print("Starting split task list")
    t = time()
    await split_tasks(tasks, 1)
    print("Time Taken: {}".format(time() - t))

I have defined a split tasks function as follows:

async def split_tasks(tasks, n):
    no_tasks = len(tasks)
    tasks_complete = 0
    complete = (no_tasks == tasks_complete)
    while not complete:
        tasks_to_execute = min(n, no_tasks - tasks_complete)
        task_objs_to_execute = tasks[tasks_complete:tasks_complete + tasks_to_execute]
        results = await asyncio.gather(*task_objs_to_execute, return_exceptions=True)
        tasks_complete += tasks_to_execute
        complete = (no_tasks == tasks_complete)
    return results

I have tried to execute the list of tasks twice using the split tasks function above, once with n set to 10 and once with n set to 1. My intention in the first test was for all tasks to execute concurrently and in the second for them to execute in sequence. However, it seems that all tasks are executing concurrently in both cases.

I am expecting the first function call to take about 9 seconds, and the second to take about 45 seconds (1 + 2 + ... + 9) with delays between the print statements.

EoinM
  • 1
  • 1

0 Answers0