Library multiprocess leaks named resources.

@pitrou

BPO	46391
Nosy	@pitrou, @benjaminp, @1st1, @applio, @arhadthedev, @jxdabc, @BarkingBad
PRs	gh-90549: Fix leak of global named resources using multiprocessing spawn #30617
Files	screen.png: screenshot

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-01-15.18:04:57.278>
labels = ['3.8', '3.9', '3.10', 'performance', '3.11', 'library']
title = 'Library multiprocess leaks named resources.'
updated_at = <Date 2022-02-18.19:39:09.294>
user = 'https://github.com/jxdabc'

bugs.python.org fields:

activity = <Date 2022-02-18.19:39:09.294>
actor = 'BarkingBad'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-01-15.18:04:57.278>
creator = 'milestonejxd'
dependencies = []
files = ['50580']
hgrepos = []
issue_num = 46391
keywords = ['patch']
message_count = 5.0
messages = ['410654', '410657', '410853', '411456', '413504']
nosy_count = 8.0
nosy_names = ['pitrou', 'benjamin.peterson', 'sbt', 'yselivanov', 'davin', 'arhadthedev', 'milestonejxd', 'BarkingBad']
pr_nums = ['30617']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue46391'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

Repo is the standard tool that Google uses to build Android, Chrome OS, Chromium, etc, written in Python.

Many Repo users have encountered resource leak warnings with Python 3.9.
https://bugs.chromium.org/p/gerrit/issues/detail?id=14934&q=component%3Arepo&can=5

I did some work and found that the problem is not caused by the code of Repo, but a bug of the Python library, multiprocess.

To make it simple, the Python script below leaks named resource even when exiting normally. (And be unlinked by resource_tracker with a warning. )

import multiprocessing as mp

global_resource = mp.Semaphore()

def submain(): pass

if __name__ == '__main__':
    p = mp.Process(target=submain)
    p.start()
    p.join()

Tested on macOS with Python 3.9.7

python test.py
resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown.

This bug will 100% reproduce when then main module uses named resources as global variables and uses spawn context, which is the case of Repo on macOS.

This is caused by multiprocess::BaseProcess::_bootstrap.

When a new process is started with multiprocessing.Process.start() in spawn context.

The main module is reloaded in the subprocess (for pickle) in multiprocessing::spawn::_main.
Named resources (such as the semaphore above) in the main module resister their _cleanup into multiprocessing::util::_finalizer_registry, which unlink themselves.
multiprocess::BaseProcess::_bootstrap then clears _finalizer_registry.

When a subprocess is spawned, it is no need to clear util::_finalizer_registry (and no need to call util::_run_after_forkers). Disable clearing _finalizer_registry (and disable call to _run_after_forkers) should fix this bug without breaking anything else.

And I uploaded a MR.

I added core devs related to multiprocessing into a nosy list so they got a notification and the PR will be evaluated and merged faster. FYI, the devs are Davin Potts and Antoine Pitrou (as per <https://devguide.python.org/experts/\>).

loop file owner.

update,

Confirmed to affect all versions above 3.8+.

It's really annoying.

Command line tools like Repo get a warning after every (success) command.
And mess up with shell.

-------------------------------------------------------------------

$ repo status .
project ksbox/                                  branch dev
$ /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

$
$
$
$ repo sync .
Fetching: 100% (1/1), done in 1.184s
Garbage collecting: 100% (1/1), done in 0.013s
repo sync has finished successfully.
$ /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

$
$
$
$ repo status .
project ksbox/ branch dev
$ /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

gh-90549: Fix leak of global named resources using multiprocessing sp…

pythongh-90549: Fix leak of global named resources using multiprocess…

gh-90549: Fix leak of global named resources using multiprocessing sp…

@jxdabc

Okay, this is fixed now. Thanks @jxdabc for the contribution!

Revert 30935 defer bpo45162 to 312 (#94285)

Which version(s) was this fix applied to? I'm still getting this error in the latest version of Python 3.11.4 when running repo status

/Users/donturner/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

This is also happening in 3.11.5

/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Has this been resolved? Issue on macOS running python 3.9.12.

3.10.6+ and 3.11.0+ contain PR #30617. The reproducer code listed in the opening message does not cause the Resource warning for me (including when explicitly setting the start method to 'spawn'). If you're still seeing this on recent Python versions, we need a simple way to reproduce it.

I recommend opening a new issue when you have one.

(Just because you see this error message and find your way here does not mean that Python itself is the cause -- it can also be an application bug, we currently lack sufficient information to say one way or the other.)

fix: clean shutdown — dispose database engine and suppress resource_t…

fix: suppress benign multiprocessing semaphore warning on shutdown

Navigation Menu

Repository navigation

Library multiprocess leaks named resources. #90549

16 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Library multiprocess leaks named resources. #90549

Description

Activity

jxdabc commented on Jan 15, 2022

arhadthedev commented on Jan 15, 2022

jxdabc commented on Jan 18, 2022

jxdabc commented on Jan 24, 2022

16 remaining items

pitrou commented on Jun 10, 2022

dturner commented on Aug 17, 2023

siennathesane commented on Sep 11, 2023

3DMiller commented on Mar 4, 2024

gpshead commented on Mar 4, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions