67

Is there any easy way to have a system-wide mutex in Python on Linux? By "system-wide", I mean the mutex will be used by a group of Python processes; this is in contrast to a traditional mutex, which is used by a group of threads within the same process.

EDIT: I'm not sure Python's multiprocessing package is what I need. For example, I can execute the following in two different interpreters:

from multiprocessing import Lock
L = Lock()
L.acquire()

When I execute these commands simultaneously in two separate interpreters, I want one of them to hang. Instead, neither hangs; it appears they aren't acquiring the same mutex.

Martin Thoma
  • 108,021
  • 142
  • 552
  • 849
emchristiansen
  • 3,368
  • 3
  • 24
  • 39
  • http://stackoverflow.com/questions/5756813/simple-but-fast-ipc-method-for-a-python-and-c-application – Anycorn Aug 03 '11 at 18:37

6 Answers6

39

The "traditional" Unix answer is to use file locks. You can use lockf(3) to lock sections of a file so that other processes can't edit it; a very common abuse is to use this as a mutex between processes. The python equivalent is fcntl.lockf.

Traditionally you write the PID of the locking process into the lock file, so that deadlocks due to processes dying while holding the lock are identifiable and fixable.

This gets you what you want, since your lock is in a global namespace (the filesystem) and accessible to all processes. This approach also has the perk that non-Python programs can participate in your locking. The downside is that you need a place for this lock file to live; also, some filesystems don't actually lock correctly, so there's a risk that it will silently fail to achieve exclusion. You win some, you lose some.

Gary van der Merwe
  • 8,627
  • 3
  • 44
  • 77
zmccord
  • 2,259
  • 16
  • 14
  • 1
    Logical place for the lock file is `/var/lock` - however if there are going to be a vast number of locking operations, I suggest `/tmp` as not all systems have `/var/lock` in `tmpfs` ramdisk. – Kimvais Feb 27 '12 at 12:27
  • Not all systems have /tmp in tmpfs ramdisk either; my install of OS X does not seem to. Good points all the same, though. – zmccord Feb 27 '12 at 13:03
  • he was asking about _linux_ and most (if not all) of the major modern linux distros have /tmp in /tmpfs - none have /var/lock by default IIRC. – Kimvais Feb 27 '12 at 13:16
  • My Debian Squeeze install doesn't have a tmpfs /tmp either. It's actually got tmpfs on /dev/shm, which feature I've also seen on Fedora. It probably doesn't much matter in the end, as long as the location is documented. – zmccord Mar 01 '12 at 15:11
  • 1
    Which filesystems don't lock correctly? Or, where can I find info on which filesystems don't lock correctly? – cowlinator Jan 11 '18 at 20:02
  • `write the PID of the locking process into the lock file` That's very difficult to do since w+ (read,write,truncate if exists, create if doesn't exist) truncates the file before the locking attempt, and a+ (read,append_write,create if doesn't exist) doesn't allow seek(0) on some Linux's – Ben Slade Mar 25 '22 at 17:28
  • https://stackoverflow.com/questions/29013495/opening-file-in-append-mode-and-seeking-to-start addresses these issues. – zmccord Mar 26 '22 at 20:27
15

Try ilock library:

from ilock import ILock

with ILock('Unique lock name'):
    # The code should be run as a system-wide single instance
    ...
Symon
  • 1,428
  • 18
  • 30
  • ... or directly via `portalocker`: `with portalocker.TemporaryFileLock('filename.lock'): ...` – KT. May 12 '18 at 20:16
  • @KT. TemporaryFileLock has race issues. Also, ILock support re-entrance. – Symon May 16 '18 at 19:01
  • Are you sure about the race issues? It seems to me both `ILock` and `portalocker.Lock` use exactly the same strategy of first doing `open` and then simply calling `portalocker.lock` on the handle to gain exclusivity. Where would the latter implementation miss anything to cause a race issue? – KT. May 16 '18 at 19:52
  • 2
    There can be an unexpected exception raised in case when file is deleted – Symon May 20 '18 at 16:13
  • I see, you are referring to the "WindowsError" catching piece in your code. It would probably make sense to add this try-catch to portalocker as well then. If you then also add a TemporaryFileRLock (which is a matter of 10 lines), the need for maintaining a separate library would probably go away. – KT. May 21 '18 at 13:36
  • @KT. With portalocker, can I specify whether the code should block if the lock is taken or raise an Exception? Is there a simple API for using the lock without a context manager? And can I specify non-exclusive read locks as well as exclusive write locks? – Konstantin Schubert Jul 21 '18 at 14:40
  • @KonstantinSchubert all the docs are here: http://portalocker.readthedocs.io/en/latest/ – KT. Jul 22 '18 at 21:03
12

The POSIX standard specifies inter-process semaphores which can be used for this purpose. http://linux.die.net/man/7/sem_overview

The multiprocessing module in Python is built on this API and others. In particular, multiprocessing.Lock provides a cross-process "mutex". http://docs.python.org/library/multiprocessing.html#synchronization-between-processes

EDIT to respond to edited question:

In your proof of concept each process is constructing a Lock(). So you have two separate locks. That is why neither process waits. You will need to share the same lock between processes. The section I linked to in the multiprocessing documentation explains how to do that.

wberry
  • 17,347
  • 8
  • 51
  • 82
  • Thanks, but "multiprocessing" doesn't appear to be what I need; see the edited question. – emchristiansen Aug 03 '11 at 19:51
  • 28
    The section you linked to appears to show how a master process can spawn 10 processes, passing a Lock object to each one it creates. My use case is different, as there is no master process spawning subprocesses. In my case, each process is invoked completely independently, but they must still coordinate. – emchristiansen Aug 05 '11 at 01:54
  • 3
    Shared memory with a configured numeric address may be the only option if there is no relationship between peers but they still need a shared mutex. The mutex object can then live in the shared memory segment. There may be no Python API for this; if not you may have to go native. Also confirm that PThreads supports this use case fully; I worry it may not. Also, to me this is a design smell; it seems like you should be either using threads and mutexes, or a separate process like redis or riak to arbitrate. – wberry Sep 09 '14 at 15:11
11

My answer overlaps with the other answers, but just to add something people can copy-paste, I often do something like this.

class Locker:
    def __enter__ (self):
        self.fp = open("./lockfile.lck")
        fcntl.flock(self.fp.fileno(), fcntl.LOCK_EX)

    def __exit__ (self, _type, value, tb):
        fcntl.flock(self.fp.fileno(), fcntl.LOCK_UN)
        self.fp.close()

And then use it as:

print("waiting for lock")
with Locker():
    print("obtained lock")
    time.sleep(5.0)

To test, do touch lockfile.lck then run the above code in two or more different terminals (from the same directory).

Keeely
  • 695
  • 8
  • 17
  • Any specific reason why you use `"rb"` and not `"wb"`? – theV0ID Jul 12 '20 at 17:35
  • @theV0ID flock doesn't require write access so there's no need to request it. In fact the 'rb' is not required, so I've removed it thanks for the question! – Keeely Jul 15 '20 at 10:26
  • But by using `"wb"` you would omit the necessity to run `touch lockfile.lck` on a terminal first, right? – theV0ID Jul 15 '20 at 10:50
  • 1
    I tend to want the file created first, generally by the super-user so the individual workers can't delete it. I also want to avoid something with greater privileges coming and writing the file before any workers start, potentially denying them access. I consider the job of creating that file down to my 'installer' (for want of a better word), and my preference is to see any problems with that file creation at install time instead of run-time when it's hard to debug. But if it's right for your application, go for it! – Keeely Jul 15 '20 at 11:29
  • This seems specific to Unix-like system. – smwikipedia Apr 27 '22 at 09:48
  • @smwikipedia The question was Linux-specific. On Windows you have a global namespace for mutex objects so you would use CreateMutex to create a named mutex and then wait on it with WaitForSingleObject. That's what I remember from my C++ days and I assume these functions are available from the Python win32 extensions (although I've never needed this in Python so far). – Keeely Apr 27 '22 at 10:36
5

Just to add a one to the list, there's the posix_ipc library, which has a Semaphore class.

A Semaphore with a count of 1 can be used as a Mutex. To complete the threading trio, the SystemEvent library makes use of posix_ipc and provides an Event as well.

I'd also note that this doesn't poll your hard drive either!

c z
  • 6,327
  • 3
  • 35
  • 50
  • 1
    From a different source I also found this library as most useful for the origial problem. In my case: 2-3-4 python programs (the same progam, started 2-3-4 times with different parameters) - and they have to use a non-thread safe resource. POSIX_IPC was the easiest solution by far. No file-system affected things are needed. – V-Mark Jan 13 '20 at 20:42
1

For a system-wide mutex that enables the synchronization of absolutely separate processes (i.e., to INCLUDE Linux processes that do NOT belong to the same processes tree), simply use fcntl.flock. I suppose that using a memory file under Linux' /run/shm folder may make it perform faster.

See more here.

Ofer
  • 385
  • 5
  • 11