Issue 30556: asyncio.wait very slow with FIRST_COMPLETED

➜

This issue tracker will soon become read-only and move to GitHub.
For a smoother transition, remember to log in and link your GitHub username to your profile.
For more information, see this post about the migration.

classification

Title:	asyncio.wait very slow with FIRST_COMPLETED
Type:	performance	Stage:
Components:	asyncio	Versions:	Python 3.6

process

Status:	open	Resolution:
Dependencies:		Superseder:
Assigned To:		Nosy List:	asvetlov, samuelcolvin, yselivanov
Priority:	normal	Keywords:

Created on 2017-06-02 17:47 by samuelcolvin, last changed 2022-03-05 06:57 by kumaraditya303.

Files
File name	Uploaded	Description	Edit
asyncio_wait_test.py	samuelcolvin, 2017-06-02 17:47

Messages (1)
msg295045 - (view)	Author: Samuel Colvin (samuelcolvin) *	Date: 2017-06-02 17:47
This is best described the script at https://gist.github.com/samuelcolvin/00f01793c118bf9aafae886ffbc81a58. Basically, completing a set of tasks using asyncio.wait(...return_when=asyncio.FIRST_COMPLETED) in a loop to wait for them to finish is much much slower than completing the same set of tasks using just asyncio.wait(pending_tasks, loop=loop, return_when=asyncio.ALL_COMPLETED). In my example, the ALL_COMPLETED case takes ~5s, but the FIRST_COMPLETED case takes 18 - 47s. The screenshot on the gist shows network usage with a very long tail, in other words very few tasks are taking a long time to complete. This example uses aiohttp get a url as the test case but I'm pretty sure this isn't an issue with aiohttp. I tried the same thing with uvloop but got broadly the same result.

msg295045 - (view)

Author: Samuel Colvin (samuelcolvin) *

Date: 2017-06-02 17:47

This is best described the script at https://gist.github.com/samuelcolvin/00f01793c118bf9aafae886ffbc81a58.

Basically, completing a set of tasks using asyncio.wait(...return_when=asyncio.FIRST_COMPLETED) in a loop to wait for them to finish is much much slower than completing the same set of tasks using just asyncio.wait(pending_tasks, loop=loop, return_when=asyncio.ALL_COMPLETED).

In my example, the ALL_COMPLETED case takes ~5s, but the FIRST_COMPLETED case takes 18 - 47s.

The screenshot on the gist shows network usage with a very long tail, in other words very few tasks are taking a long time to complete.

This example uses aiohttp get a url as the test case but I'm pretty sure this isn't an issue with aiohttp.

I tried the same thing with uvloop but got broadly the same result.

History
Date	User	Action	Args
2022-03-05 06:57:40	kumaraditya303	set	nosy: + asvetlov
2017-06-02 17:47:13	samuelcolvin	create