Error: "Connection pool is full"

[andreasjansson]

When parallelizing requests with asyncio (wrapping replicate.run in asyncio.get_running_loop().run_in_executor I get

WARNING - Connection pool is full, discarding connection: api.replicate.com. Connection pool size: 10

Having an async client might fix this (see #74).

[mattt]

Hi @andreasjansson. Async support for the Replicate client library just landed in version 0.18.0. When you have an opportunity, can you try replacing run with async_run in your code, and let me know whether that resolved your issue?

Here's an example of how to run multiple predictions concurrently and wait for results:

import asyncio
import replicate

# https://replicate.com/stability-ai/sdxl
model_version = "stability-ai/sdxl:39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b"
prompts = [
    f"A chariot pulled by a team of {count} rainbow unicorns"
    for count in ["two", "four", "six", "eight"]
]

async with asyncio.TaskGroup() as tg:
    tasks = [
        tg.create_task(replicate.async_run(model_version, input={"prompt": prompt}))
        for prompt in prompts
    ]

results = await asyncio.gather(*tasks)
print(results)