0

When i run the multiprocessing example on a OSX. I get the Error OSError: [Errno 28] No space left on device.

The ENOSPC ("No space left on device") error will be triggered in any situation in which the data or the metadata associated with an I/O operation can't be written down anywhere because of lack of space. This doesn't always mean disk space – it could mean physical disk space, logical space (e.g. maximum file length), space in a certain data structure or address space. For example you can get it if there isn't space in the directory table (vfat) or there aren't any inodes left. It roughly means “I can't find where to write this down”. Source: https://stackoverflow.com/a/6999259/330658

What i don't understand, where files are written down in my code below?

Any help highly appreicated.

Example Code:

#! /usr/bin/env python3
import sys
import os
import multiprocessing as mp
import time


def foo_pool(x):
    time.sleep(2)
    return x*x

result_list = []
def log_result(result):
    result_list.append(result)
    print(result)

def apply_async_with_callback():
    pool = mp.Pool()
    for i in range(10):
        pool.apply_async(foo_pool, args = (i, ), callback = log_result)
    pool.close()
    pool.join()
    print(result_list)

if __name__ == '__main__':
    apply_async_with_callback()

Full Error:

python3 test.py
Traceback (most recent call last):
  File "test.py", line 32, in <module>
    apply_async_with_callback()
  File "test.py", line 23, in apply_async_with_callback
    pool = mp.Pool()
  File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 191, in __init__
    self._setup_queues()
  File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 343, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
  File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/queues.py", line 342, in __init__
    self._rlock = ctx.Lock()
  File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
OSError: [Errno 28] No space left on device
endo.anaconda
  • 2,364
  • 4
  • 27
  • 54
  • Hmm. Works fine when I run it on my Mac. The error message makes it look like your program is having problems creating the pool in the first place. – Frank Yellin Nov 30 '21 at 21:06
  • https://stackoverflow.com/questions/43573500/no-space-left-while-using-multiprocessing-array-in-shared-memory looks like it is relevant. – Frank Yellin Nov 30 '21 at 21:13
  • as far as I remember, semaphores create some files in procfs. It is not a real filesystem on an HDD/SSD - so, yes, the error is misleading. The number of semaphores the kernel can create is configurable, but I don't see how this code can exhaust all available semaphores. Is there anything special with the target system - e.g., does it happen to run a loaded database or has custom kernel parameters? – Marat Nov 30 '21 at 21:38
  • On Linux, there is the environment variable `TMPDIR` to define where the shared memory objects of the `multiprocessing` library go to. I`ve set this to a large enough dir on a ram device. – Nechoj Nov 30 '21 at 22:05

0 Answers0