56

I'm trying to learn how to use Python's multiprocessing package, but I don't understand the difference between map and imap.

Is the difference that map returns, say, an actual array or set, while imap returns an iterator over an array or set? When would I use one over the other?

Also, I don't understand what the chunksize argument is. Is this the number of values that are passed to each process?

grautur
  • 29,127
  • 34
  • 92
  • 127
  • 5
    Closely related: [multiprocessing.pool: What's the difference between map_async and imap?](http://stackoverflow.com/questions/26520781/multiprocessing-pool-whats-the-difference-between-map-async-and-imap/26521507#26521507) – dano Jan 28 '16 at 19:34

3 Answers3

49

That is the difference. One reason why you might use imap instead of map is if you wanted to start processing the first few results without waiting for the rest to be calculated. map waits for every result before returning.

As for chunksize, it is sometimes more efficient to dole out work in larger quantities because every time the worker requests more work, there is IPC and synchronization overhead.

Antimony
  • 35,689
  • 10
  • 93
  • 100
  • 1
    So how does one approach determining a reasonable value for chunksize then? If bigger means less IPC & sync overhead due to pickling, what's the tradeoff? (ie why is picking `chunksize == len(iterable)` a bad idea, or is it?) – Adam Parkin Aug 24 '12 at 17:54
  • 2
    @Adam If you pick `chunksize = len(iterable)`, then all the jobs will be assigned to a single process! `len(iterable) // numprocesses` is the maximum that is useful. The tradeoff is between synchronization overhead and cpu utilization (large chunksizes will cause some processes to finish before others, wasting potential processing time). – Antimony Oct 03 '12 at 18:27
  • Ok, I see that, but that simply mean picking a reasonable chunksize boils down to trial and error on particular data in a particular setting? – Adam Parkin Oct 03 '12 at 18:29
  • I think so. Most optimization requires profiling and fine tuning. – Antimony Oct 03 '12 at 22:34
  • 1
    It's also worth mentioning that imap can be applied to a generator, while map will turn your generator into a list-like object, so imap doesn't wait for the input to get generated. – mgoldwasser Dec 08 '15 at 21:07
  • Does that mean that `imap` returns the first result from the x number of processes running? i.e. does it retain order? Does that mean that you still have to wait for the first process to finish before you can start the iteration on the results? – Tjorriemorrie Jan 13 '16 at 13:31
4

imap is from itertools module which is used for fast and memory efficiency in python.Map will return the list where as imap returns the object which generates the values for each iterations(In python 2.7).The below code blocks will clear the difference.

Map returns the list can be printed directly

 from itertools import *
    from math import *

    integers = [1,2,3,4,5]
    sqr_ints = map(sqrt, integers)
    print (sqr_ints)

imap returns object which is converted to list and printed.

from itertools import *
from math import *

integers = [1,2,3,4,5]
sqr_ints = imap(sqrt, integers)
print list(sqr_ints)

Chunksize will make the iterable to be split into pieces of specified size(approximate) and each piece is submitted as a separate task.

Chandan
  • 610
  • 6
  • 17
  • 2
    While the spirit of this answer is correct, notice that the original question is asking about **multiprocessing** map and imap. – Jacob Jones Jun 27 '20 at 22:54
3

With imap, forked calls are done in parallel, not one after another sequentially. For example, below you're hitting say three exchanges to get order books. Instead of hitting exchange 1, then exchange 2, then exchange 3 sequentially, imap.pool calls are non-blocking and goes straight to all three exchanges to fetch order books as soon as you call.

from pathos.multiprocessing import ProcessingPool as Pool
pool = Pool().imap
self.pool(self.getOrderBook, Exchanges, Tickers)