httpx AsyncClient pool limits keepalive tuning

abstract

A teammate paged me at 11pm on a Friday: "we're seeing ReadTimeout on the billing route again." We'd already raised max_connections twice that month, CPU was sitting at 12%, and the pool wasn't even close to full. We were losing requests to something none of us had thought to look at. The culprit was the keepalive_expiry value httpx had been quietly using since the day we installed it. I won't promise this article makes httpx pool tuning trivial; it isn't. By the end you'll know which of three knobs to turn for which symptom, and you'll have five working commits you can replay locally to feel each fix land.

Step 1: The implicit defaults are not "no limits" (commit 38fd8df)

Most people read httpx.AsyncClient() and assume it has no limits set. It has three of them, and one is short enough to bite you at the first traffic spike. httpx applies DEFAULT_LIMITS = Limits(max_connections=100, max_keepalive_connections=20, keepalive_expiry=5.0) quietly. That number has been stable since 0.18 and ships from httpx._config. You can confirm it yourself:

from httpx._config import DEFAULT_LIMITS

print(DEFAULT_LIMITS.max_connections)            # 100
print(DEFAULT_LIMITS.max_keepalive_connections)  # 20
print(DEFAULT_LIMITS.keepalive_expiry)           # 5.0

100 simultaneous open sockets and 20 idle ones held for 5 seconds. That works for a low-traffic CRUD app where the upstream is one origin, idle timeouts are generous, and bursts top out below 20 concurrent calls. It breaks the moment any of those assumptions slip.

Why 5 seconds for keepalive_expiry? Because nginx's default keepalive_timeout is 75 seconds, and most application origins sit between 30 and 60. 5.0 is a conservative floor that fits inside almost any origin's idle window. The catch is that the typical origin is not the typical PROXY. AWS API Gateway, Cloudflare, and a default Envoy front line close idle TCP much sooner than the upstream behind them, and your pool happily hands back a socket those proxies already reaped.

The companion pool_demo.baseline module captures exactly this shape:

from pool_demo.baseline import implicit_default_limits, describe_limits

limits = implicit_default_limits()
print(describe_limits(limits))  # (100, 20, 5.0)

The point of having a control module is that every later lesson reuses these numbers as the "what the framework did when you said nothing" benchmark.

Step 2: What the three knobs actually do (commit a2a5866)

Each of the three knobs has a one-sentence description in the docs. The way they interact has none, and that gap is where most pool incidents start.

max_connections is the hard ceiling on simultaneous open TCP sockets per pool. When the pool hits this number, the next await client.get blocks on a semaphore until a slot frees. Hitting this is a SYMPTOM, not a root cause. The two failure modes that get you here are (a) traffic outgrowing the pool and (b) upstream latency stretching request lifetime. Raising the number fixes (a) but not (b); for (b) you need to fix the upstream.

max_keepalive_connections is the soft ceiling on idle sockets the pool holds open between requests. When a request completes, its connection is returned to the pool. If the pool already holds max_keepalive_connections idle sockets, the new returnee is closed immediately. Set this lower than max_connections and every overflow request pays the cold-TCP price (a 3-way handshake plus TLS) instead of reusing a warm socket.

keepalive_expiry is seconds an idle socket may sit in the pool before the pool itself closes it. 5.0 is the default. This is the knob that interacts with proxies; we come back to it in Step 5.

The Lesson 2 module ships an explicit constructor so the three numbers are visible at the call site, plus a saturate_pool helper that demonstrates the queue behaviour:

from pool_demo.defaults import build_limits, make_explicit_client, saturate_pool

limits = build_limits(
    max_connections=3,
    max_keepalive_connections=2,
    keepalive_expiry=1.0,
)
async with make_explicit_client(transport, limits) as client:
    trace = await saturate_pool(client, "https://example.test/items", burst=6, max_connections=3, per_request_latency=0.05)
print(trace.wall_seconds, trace.statuses_ok)

MockTransport does not actually impose pool blocking, so the test asserts the trace shape rather than the wall time. Against a real socket-backed transport with a 50ms upstream and max_connections=3, six concurrent requests serialise into two waves of three and finish in roughly 100ms instead of the 50ms they would take with max_connections=6. That is the cost of capping too low.

Step 3: Size the pool from the workload, not from a habit (commit b8e81f0)

How many connections does your pool actually need? One line of arithmetic answers it:

target_concurrency      = expected_qps * p99_latency_seconds
max_connections         = ceil(target_concurrency * safety_factor)
max_keepalive_connections = ceil(target_concurrency)
keepalive_expiry        = min(upstream_idle_timeout, lb_idle_timeout)

expected_qps is sustained requests per second to THIS upstream, not the whole service. p99_latency_seconds is the P99 you measure in production (not the median; the median underestimates concurrency at the tail). safety_factor defaults to 1.5 because that is roughly one P99 standard deviation of headroom.

The pool_demo.tuned module ships an UpstreamSpec dataclass and a size_limits function that runs that arithmetic:

from pool_demo.tuned import UpstreamSpec, size_limits

slow_billing = UpstreamSpec(
    name="slow-billing",
    expected_qps=20.0,
    p99_latency_seconds=0.800,
    upstream_idle_timeout=30.0,
    lb_idle_timeout=60.0,
)
limits = size_limits(slow_billing)
# Limits(max_connections=24, max_keepalive_connections=16, keepalive_expiry=30.0)

Sized this way, three very different upstreams from one production service look like this:

Upstream	QPS	P99	LB idle	max_conn	max_keepalive	keepalive_expiry
Fast internal RPC	5000	20ms	10s	150	100	10.0
Slow billing API	20	800ms	60s	24	16	30.0
Object storage	200	60ms	30s	18	12	30.0

Three upstreams, three pool shapes. A single httpx.Limits(100, 20, 5.0) underprovisions the RPC by 5x AND wastes idle keepalive slots at the billing API. Sizing per upstream costs you one dataclass per host and gives back tail-latency stability that no global default can match.

Step 4: One AsyncClient per host, dispatched by registry (commit a3738c7)

The third anti-pattern is calling five upstreams through one shared AsyncClient. The shared client has ONE Limits shape, and when that shape is set wide enough for the slowest upstream, the fast one wastes pool slots on requests that already finished. When it is set tight enough for the fast upstream, the slow one head-of-line blocks the queue.

The fix is a thin registry: one AsyncClient per host, each sized by its own UpstreamSpec, dispatched by host at call time. The registry owns lifecycle (aclose shuts every client) and surfaces unknown hosts as KeyError so a typo in a call site fails loudly:

from pool_demo.registry import LimitsRegistry
from pool_demo.tuned import UpstreamSpec

specs = {
    "rpc.internal":   UpstreamSpec(name="rpc",     expected_qps=5000, p99_latency_seconds=0.020),
    "billing.vendor": UpstreamSpec(name="billing", expected_qps=20,   p99_latency_seconds=0.800),
    "objects.cdn":    UpstreamSpec(name="objects", expected_qps=200,  p99_latency_seconds=0.060),
}

registry = LimitsRegistry.from_specs(specs)
try:
    resp = await registry.get_for("billing.vendor").get("/charges/42")
finally:
    await registry.aclose()

The registry pattern collapses the "shared client" trap into a fail-loud constructor. A 4-line dictionary in your composition root replaces the 200-line Slack incident you would have had at 3am.

The size of LimitsRegistry is intentionally tiny (one dict, one method, one shutdown). When you reach for a feature like rotating credentials, retry policy, or tracing, push it onto the specific AsyncClient you fetch from get_for, not into the registry itself. The registry's job is to keep one pool per upstream; everything else is the call site's concern.

Step 5: Behind an aggressive load balancer (commit fe4b8b8)

The final case is the one most operators discover after a Friday-evening deploy. Your upstream is healthy. Your pool is sized correctly. You start seeing RemoteProtocolError: Server disconnected without sending a response on roughly 0.4% of requests. The number is small enough that retries hide it, large enough that PagerDuty notices.

The cause is a keepalive_expiry that exceeds the load balancer's idle timeout. Cloudflare's edge closes idle TCP at roughly 1.0 second. Default keepalive_expiry=5.0 means the pool happily hands out a connection the proxy already reaped four seconds ago. The first byte of the new request lands on a closed socket and httpx surfaces it as RemoteProtocolError.

The fix is two parts. First, clamp keepalive_expiry below the LB idle timeout (lb_idle_timeout - epsilon, where epsilon covers clock drift and the round-trip of the close handshake). 0.1 seconds is enough for most edges. Second, wrap the request in a single RemoteProtocolError retry, so the precise race between pool acquire and first byte recovers transparently:

from pool_demo.load_balancer import request_with_retry, size_limits_for_lb
from pool_demo.tuned import UpstreamSpec

edge = UpstreamSpec(
    name="edge-api",
    expected_qps=200,
    p99_latency_seconds=0.040,
    upstream_idle_timeout=30.0,
    lb_idle_timeout=1.0,
)
limits = size_limits_for_lb(edge, epsilon=0.1)
# keepalive_expiry == 0.9, max_keepalive_connections == 8, max_connections == 12

async with httpx.AsyncClient(transport=transport, limits=limits) as client:
    resp = await request_with_retry(client, "GET", "https://edge-api.example.com/v1")

A second RemoteProtocolError after retry means something other than the LB-keepalive race is happening (often a half-closed socket from the upstream itself) so it is correct to surface that error instead of looping. One retry is the difference between "0.4% error rate" and "a Slack message that nobody acts on".

The request_with_retry helper also illustrates a more general pattern: pool tuning fixes the steady state, but a one-shot retry covers the inevitable edge between "pool acquires" and "first byte sent". You want both. Tuning without retry leaves a small error rate visible to clients; retry without tuning hides the leak under a doubled request count that eventually saturates the pool.

When to use which knob

Across the five lessons, the decision tree collapses to four rules:

If your service calls more than one upstream, build a LimitsRegistry. The shared-client trap is the single biggest source of production tail-latency regressions in httpx-based services.
Size max_keepalive_connections to your steady-state P99 concurrency, not to round numbers. 20 is the default; it is rarely the right answer.
Set keepalive_expiry to min(upstream_idle_timeout, lb_idle_timeout) - 0.1. The "minus 0.1" is the cheapest insurance you can buy against RemoteProtocolError.
Wrap requests behind a flaky edge in a single RemoteProtocolError retry. Not three retries; one. The race window is short, and longer loops mask real upstream failures.

Repository

Full source at https://github.com/vytharion/httpx-asyncclient-pool-limits-keepalive.

Lesson 1: 38fd8df, implicit-defaults baseline.
Lesson 2: a2a5866, explicit Limits plus saturation trace.
Lesson 3: b8e81f0, size_limits workload formula.
Lesson 4: a3738c7, per-host LimitsRegistry.
Lesson 5: fe4b8b8, aggressive-LB keepalive_expiry plus retry.

Clone, run uv sync then uv run pytest -q, and git checkout <sha> to step through any commit on its own.

Closing

The pool is rarely the first place anyone looks when a service starts dropping requests, and that is part of the problem. The five commits in this repo flip that order: every change is one number, every number ties to a specific workload property (QPS, P99, LB timeout), and every fix is testable without leaving your laptop. Clone the repo, run the tests at each commit, and the next time a burst arrives you will know exactly which knob to turn.