How does one use multiprocessing.Pool with a target function that either returns a generator or is a generator function? Below is an example.
def sample_function(i):
return (i+j for j in range(10))
with multiprocessing.Pool(processes=5) as pooled_tasks:
results = pooled_tasks.imap_unordered(sample_function,range(100))
pooled_tasks.close()
pooled_tasks.join()
return_values = []
# results should be a list of lists
# result should be a list of generators
for result in results:
return_values += result
# return_values should be a list of all generators returned
print(return_values)
But this raises the following error:
multiprocessing.pool.MaybeEncodingError: Error sending result: '<generator object sample_function.<locals>.<genexpr> at 0xabcabcabcac>'. Reason: 'TypeError("cannot pickle 'generator' object")'
How does one modify the multiprocessing to be able to handle generators?
I'd also be interested if there's a better solution that expands return_values from a list of generators to a list of lists containing generated values, without changing sample_function.
