A per-worker concurrency limit tells the gateway never to route more than a set number of in-flight requests to a box <a href="#ref-1">[1]</a>. A saturated worker is skipped, and a fully saturated pool answers <code>503</code> with a <code>Retry-After</code> header immediately. That is honest backpressure rather than a queue that grows silently until something falls over.
Keep each box in its sweet spot
A box asked to run far more concurrent generations than it can handle does not degrade gracefully; it thrashes, runs out of memory, or takes its neighbours down with it <a href="#ref-1">[1]</a>. A cap holds each box at the concurrency where its latency is still acceptable. On a mixed fleet you set a higher cap on the big box and a lower one on a small box, and each contributes what it actually can.
Fast-fail beats slow-death
When the pool is full, callers are told to retry shortly rather than left hanging until a timeout <a href="#ref-1">[1]</a>. AI Suite apps honour <code>Retry-After</code> automatically and fall back to on-device AI where it is available, so a busy morning degrades into a brief wait instead of a stall <a href="#ref-2">[2]</a>.
Sizing the cap
Choose a cap by starting at the number of concurrent generations the box handles at acceptable latency and watching the worker card's latency on the dashboard as load rises; the knee is obvious <a href="#ref-2">[2]</a>. A cap is hard: a saturated worker gets nothing, not even failover traffic, which is what protects it from being pushed past the edge at the worst moment. A cap of zero means uncapped.