1

I have a thread in my python program that acquires images from a webcam and puts them in a multiprocessing queue. A separate process then takes these images from the queue and does some processing. However, if I try to empty the queue from the image acquisition (producer) thread I do not free any memory, and the program eventually uses all the available memory and crashes the machine (Python 3.6.6 / Ubuntu 18.04 64bit / Linux 4.15.0-43-generic)

I have a simple working example that reproduces the problem.

import multiprocessing
import time
import numpy as np

queue_mp = multiprocessing.Queue(maxsize=500)

def producer(q):
    while True:
        # Generate object to put in queue
        dummy_in = np.ones((1000,1000))

        # If the queue is full, get the oldest object (FIFO),
        # to make space for the latest incoming object.
        if q.full():
            __ = q.get()
        q.put(dummy_in)


def consumer(q):
    while True:
        # Get object from queue
        dummy_out = q.get()

        # Do some processing on the object, which we simulate here by time.sleep
        time.sleep(3)

producer_process = multiprocessing.Process(target=producer,
                                           args=(queue_mp,),
                                           daemon=False)

consumer_process = multiprocessing.Process(target=consumer,
                                           args=(queue_mp,),
                                           daemon=False)

# Start producer and consumer processes
producer_process.start()
consumer_process.start()

I can rewrite my code to avoid this problem, but I'd like to understand what is happening. Is there a general rule that producers and consumers of a multiprocessing queue must be running in separate processes?

If anyone understands why this happens, or what exactly is happening behind the scenes of multiprocessing queues that would explain this memory behavior I would appreciate it. The docs did not go into a lot of detail.

nstelzen
  • 11
  • 3

1 Answers1

0

I figured out what was happening, so I'll post it here for the benefit of anyone that stumbles across question.

My memory problem resulted from a numpy bug in numpy version 1.16.0. Reverting to numpy version 1.13.3 resolved the problem.

To answer the basic question: No, there is no need to worry which thread/process is doing the consuming (get) and which thread/process is doing the producing (put) for multiprocessing queues. There is nothing special about multiprocessing queues with respect to garbage collection. As kindall explains in response to a similar question:

When there are no longer any references to an object, the memory it occupies is freed immediately and can be reused by other Python objects

I hope that helps someone. In any case, the numpy bug should be resolved in the 1.16.1 release.

nstelzen
  • 11
  • 3