Question

Why is asyncio.gather slower than sequential requests in this script?

I expected these requests to run concurrently, but the async version is slightly slower.

async def load(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.json()

results = await asyncio.gather(*(load(url) for url in urls))

There are about 80 URLs from the same API.

posted by Ava Thompson · 374 views

1 answer

You create a new ClientSession and connection pool for every URL. Share one session and bound concurrency so you reuse connections without overwhelming the service.

semaphore = asyncio.Semaphore(10)

async with aiohttp.ClientSession() as session:
    async def load(url):
        async with semaphore, session.get(url) as response:
            response.raise_for_status()
            return await response.json()

    results = await asyncio.gather(*(load(url) for url in urls))

Accepted answerposted by Omar Haddad

Why is asyncio.gather slower than sequential requests in this script?

1 answer

Your response