- How can
multiprocessing.Pool be used in PyCharm Python Console (Scientific / Interactive Mode) without resulting in _pickle.PicklingError?
IDE
PyCharm 2021.3.2 (Professional Edition)
Build #PY-213.6777.50, built on January 27, 2022
Licensed to Trenton J. McKinney / Trenton McKinney
Subscription is active until February 27, 2022.
Runtime version: 11.0.13+7-b1751.25 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
Linux 3.10.0-1160.49.1.el7.x86_64
GC: G1 Young Generation, G1 Old Generation
Memory: 2048M
Cores: 32
Current Desktop: XFCE
Code
- The following code works in Terminal, and Jupyter Lab, but not PyCharm 2021.3.2
- This is a simple reproducible example
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
with Pool(32) as p:
print(p.map(f, range(32)))
Error
/home/tmckinney/anaconda3/bin/python3.9 /home/tmckinney/opt/pycharm-2021.3.1/plugins/python/helpers/pydev/pydevconsole.py --mode=client --port=5829
import sys; print('Python %s on %s' % (sys.version, sys.platform))
sys.path.extend(['/home/tmckinney/Documents/GitHub/hap_ingest'])
Python 3.9.7 (default, Sep 16 2021, 13:09:58)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.29.0 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 7.29.0
Python 3.9.7 (default, Sep 16 2021, 13:09:58)
[GCC 7.5.0] on linux
runfile('/home/tmckinney/Documents/GitHub/hap_ingest/py_files/multiprocessingTest.py', wdir='/home/tmckinney/Documents/GitHub/hap_ingest/py_files')
Traceback (most recent call last):
File "/home/tmckinney/anaconda3/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3444, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-2-8b3da05130fe>", line 1, in <module>
runfile('/home/tmckinney/Documents/GitHub/hap_ingest/py_files/multiprocessingTest.py', wdir='/home/tmckinney/Documents/GitHub/hap_ingest/py_files')
File "/home/tmckinney/opt/pycharm-2021.3.1/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 198, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "/home/tmckinney/opt/pycharm-2021.3.1/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/tmckinney/Documents/GitHub/hap_ingest/py_files/multiprocessingTest.py", line 10, in <module>
print(p.map(f, range(32)))
File "/home/tmckinney/anaconda3/lib/python3.9/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/home/tmckinney/anaconda3/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
File "/home/tmckinney/anaconda3/lib/python3.9/multiprocessing/pool.py", line 537, in _handle_tasks
put(task)
File "/home/tmckinney/anaconda3/lib/python3.9/multiprocessing/connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/home/tmckinney/anaconda3/lib/python3.9/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function f at 0x7f9ce1befa60>: attribute lookup f on __main__ failed
Expected Output
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961]
Research
- As noted in other documentation, there are some issues with interactive use of the
multiprocessing module in Windows. This is not the issue addressed by this question.
