0

I am trying to implement multiprocessing into python on windows environment. I know multiprocessing have to start with if __name__=='main'. But my problem is that I have a long pipeline of functions and first part of code is running as single process but output of first step which is a list needs to be run parallely as it takes lot of time. So my question is how to run output of a single process function into multiple processes.

One solution is keeping multiprocess code in some other script under main and call it using os.system from main script.

MainScript.py

import os
....
......
....
names = [('frank', 'surname'), ('justin', 'surname'), ('osi', 'surname'), ('thomas', 'surname')]
cmd = 'python fortest.py %s' % names
os.system(cmd)

MultiProcessCode.py

from multiprocessing import Pool
import time, sys


class B:
    def __init__(self):
        self.__c = C()

    def run_C(self):
        rslt = self.__c.run()
        print rslt



def unwrap_fms(arg):
    return C.f(*arg)

class C:
    def __init__(self, print_init = 'Initialized'):
        print print_init

    def f(self, name):
        return name[0]
        #print "hello %s %s" %(name[0], name[1])

    def run(self):
       pool = Pool(processes=2)
       names = [('frank', 'surname'), ('justin', 'surname'), ('osi', 'surname'), ('thomas', 'surname')]
       #print zip([self]*len(names), names)
       rslt = pool.map(unwrap_fms, zip([self]*len(names), names))
       return rslt

if __name__ == '__main__':
    b =B()
    b.run_C()

But my input list is too big, so is there a better way to do this and takes a couple of objects as well. Or if I can put things in one file itself

user3629861
  • 11
  • 2
  • I'm not quite sure what you're asking. It would be helpful if you could include some sample code that shows us what you're actually trying to do. – dano Aug 05 '14 at 14:35
  • Can you clarify exactly what you're trying to do? It looks like you're trying to process many items in a list in parallel, and `pool.map()` is the right tool for that. What is it about your existing code that isn't working for you? Perhaps some showing some examples of your input and expected output would help. – skrrgwasme Sep 29 '14 at 17:18
  • And by the way, `if __name__ == '__main__':` has nothing to do with multiprocessing. It allows you to make your script behave differently when it is imported as a module vs. run as a standalone script. See [this SO question](http://stackoverflow.com/q/419163/2615940) for more details. – skrrgwasme Sep 29 '14 at 17:19

0 Answers0