SharedMemory.unlink() can fail if called during interpreter shutdown

SharedMemory.unlink() contains a import which can fail if called during interpreter shutdown.

For example when working with SharedMemory as the buffer for a numpy array it is useful to create a wrapper object to ensure that clean up is done with the object is deleted.

#import numpy as np
from multiprocessing import shared_memory

class ShArray:
    def __init__(self):
        self.shared_memory = shared_memory.SharedMemory(create=True, size=400)
        #self.array = np.ndarray((10,10), dtype="i4", buffer=self.shared_memory.buf)

    def __del__(self):
        self.shared_memory.close()
        self.shared_memory.unlink()

a1 = ShArray()

(Commented lines show my typical usage, but are not need to demonstrate the issue)

Exception ignored in: <function ShArray.__del__ at 0x7f36eee510d0>
Traceback (most recent call last):
  File "/home/sam/git/cpython/test_shared_mem2.py", line 13, in __del__
  File "/home/sam/git/cpython/Lib/multiprocessing/shared_memory.py", line 240, in unlink
ImportError: sys.meta_path is None, Python is likely shutting down
/home/sam/git/cpython/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

This happens on 3.8 and current main, both on Linux x86-64.

Patch to follow

pythongh-91577: SharedMemory move imports out of methods

gh-91577: SharedMemory move imports out of methods (#91579)

pythongh-91577: SharedMemory move imports out of methods (pythonGH-91579

@pitrou

I'm not a supporter of multiprocessing programs relying on the GC to call __del__() finalizers. You should close resources explicitly, using with: statement if possible. But well, the change seems acceptable to me, so I merged it.

cc @pitrou

gh-91577: SharedMemory move imports out of methods (GH-91579)

Seems fine to me. with statements are not always usable.

with statements are not always usable.

So we should make them usable in more cases :-)

Even without with:, the idea is to explicitly release resources, not rely on the GC for that. The objects are not finalized in a specific order by the GC. The exact order is really complicated and depends on too many things. So it's likely that your code will fall into many corner cases. ImportError: sys.meta_path is None, Python is likely shutting down is just one example of such issue that you can get if you rely on the GC.

In PyPy, it expect that it will be even worse.

So we should make them usable in more cases :-)

I mean that with statements only work if your lifetime is lexically-defined. It's not the case for more sophisticated use cases unfortunately.

Revert 30935 defer bpo45162 to 312 (#94285)

Navigation Menu

Repository navigation

SharedMemory.unlink() can fail if called during interpreter shutdown #91577

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

SharedMemory.unlink() can fail if called during interpreter shutdown #91577

Description

Activity

vstinner commented on Jun 16, 2022

pitrou commented on Jun 16, 2022

vstinner commented on Jun 16, 2022

pitrou commented on Jun 16, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions