Repository navigation
Executing code in thread or process pools: run_in_executor example #85299
Description
Activity
aaraney commented
on Jun 26, 2020 aaraneymannequinMannequinAuthorI found an issue with the concurrent.futures.ProcessPoolExecuter() example (#3) in the asyncio event loops documentation. The call to asyncio.run(main()) should be guarded by `__name__=="__main__":`, as it sits now a RuntimeError is thrown.
https://docs.python.org/3/library/asyncio-eventloop.html#executing-code-in-thread-or-process-pools
Not getting errors on my end for python 3.8 and 3.10, could you reconfirm the error?
@slateny FYI I believe, due to some oddities of the github migration, some "mannequin" users won't necessarily receive notifications on issues unless you actually @ them, even if they show up as having previously participated in the thread. (You can safely assume that non-mannequin users are still subscribed to all the issues they were "nosy" to on BPO.)
Reacted by StanleyCertainly, thanks for pinging me, @slateny. Admittedly, the original bug report is not very descriptive, so Ill try to clarify the issue.
In the python 3.8 concurrent futures documentation the provided executor example (see below) throws a
RuntimeErrorif you copy and paste the example into a file and run it. It appears the documentation for 3.9 and 3.10 also have this issue. This is because theasyncio.run(main())statement at the bottom of the example is not guarded byif __name__ == "__main__":.import asyncio import concurrent.futures def blocking_io(): # File operations (such as logging) can block the # event loop: run them in a thread pool. with open('/dev/urandom', 'rb') as f: return f.read(100) def cpu_bound(): # CPU-bound operations will block the event loop: # in general it is preferable to run them in a # process pool. return sum(i * i for i in range(10 ** 7)) async def main(): loop = asyncio.get_running_loop() ## Options: # 1. Run in the default loop's executor: result = await loop.run_in_executor( None, blocking_io) print('default thread pool', result) # 2. Run in a custom thread pool: with concurrent.futures.ThreadPoolExecutor() as pool: result = await loop.run_in_executor( pool, blocking_io) print('custom thread pool', result) # 3. Run in a custom process pool: with concurrent.futures.ProcessPoolExecutor() as pool: result = await loop.run_in_executor( pool, cpu_bound) print('custom process pool', result) asyncio.run(main())
The example should be updated to the following:
import asyncio import concurrent.futures def blocking_io(): # File operations (such as logging) can block the # event loop: run them in a thread pool. with open('/dev/urandom', 'rb') as f: return f.read(100) def cpu_bound(): # CPU-bound operations will block the event loop: # in general it is preferable to run them in a # process pool. return sum(i * i for i in range(10 ** 7)) async def main(): loop = asyncio.get_running_loop() ## Options: # 1. Run in the default loop's executor: result = await loop.run_in_executor( None, blocking_io) print('default thread pool', result) # 2. Run in a custom thread pool: with concurrent.futures.ThreadPoolExecutor() as pool: result = await loop.run_in_executor( pool, blocking_io) print('custom thread pool', result) # 3. Run in a custom process pool: with concurrent.futures.ProcessPoolExecutor() as pool: result = await loop.run_in_executor( pool, cpu_bound) print('custom process pool', result) if __name__ == "__main__": asyncio.run(main())
Thanks for the clarification - I tried the code and it seems to run for me without error. Are you running this from another file or is this the only file?
Im running it from a standalone script. So, not from the IDLE.
Hmm interesting, I ran it on ubuntu 20 w/ python 3.8 and 3.10 and no errors for me - maybe a version/platform thing?
Reacted by Austin Raney1 remaining item
So, I was not able to reproduce the issue in a docker container. I used the following base images in attempt to be more complete:
python:3.8-slim-buster,python:3.8.10-slim-buster, andcontinuumio/miniconda3:4.11.0. However, I am beginning to think this is a platform issue not a python version issue. More specifically, something to do with the spawn semantics used vs fork exec. This behavior was changed in 3.8 for OSX.Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.
Instead of droning on about why I think that is the case, I threw together a repo that uses github actions and a macos 11 runner. Below ive included the gh action yaml for convenience. I was able to reproduce the issue using gh actions. Please find the failing job here.
name: Run Unit Tests on: push jobs: cpython_85299: runs-on: macos-11 steps: - uses: actions/checkout@v2 - name: Set up Python 3.8 uses: actions/setup-python@v2 with: python-version: '3.8' - name: Reproduce Runtime Error run: | python3 main.py
Hmm, if it's a platform issue then do you think that the docs still need updating? If the file's the only one being run, then
if __name__ == "__main__":seems equivalent to leaving it outYeah I think that is a fair point, @slateny. Ive not seen it stated in python docs that examples are runnable as their own script, but I think that is kind of implicit to an example snippet. I would prefer that
if __name__ == "__main__":is included where examples are expected to be run in a standalone context.The third example uses
ProcessPoolExecutor, defined here, which then seems to get the start method, which as you mentioned usesspawnsince 3.8. A bit down below, there is a note about protecting the program's entry point viaif __name__ == '__main__'if using thespawnmethod.Taking all that into account, I think the doc change should be instead to add a note/link to the warning for macOS, advising the entrypoint guard as needed, instead of adding it directly into the example as there might be confusion on why the guard's there when it may not be necessary.
The problem seems to be with how the OP is running the code, not with the example. Nevertheless it is fine to add a
__main__check to the example, since that is customary for all main programs. No notes about platforms though.Reacted by Austin Raney- added a commit that references this issue
on Oct 16, 2022 - added 4 commits that reference this issue
on Oct 16, 2022 - added a commit that references this issue
on Oct 17, 2022 - added a commit that references this issue
on Oct 22, 2022
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: