Real installations run mixed hardware: a big-VRAM box with a large model, a few mid-range boxes with chat models, a CPU box for embeddings. In a model-aware farm the gateway routes each request to a worker that already has the requested model warm, and advertises the union of every worker's models as one catalog <a href="#ref-1">[1]</a>. Callers see one model list and simply pick; placement is the gateway's problem, not theirs.
Warm beats cold by minutes
A model load can be a multi-gigabyte, minutes-long event. Landing a request on a box that already holds the model is the single biggest latency and efficiency win on heterogeneous farms <a href="#ref-1">[1]</a>. Requests are routed warm-first: workers that report the requested model lead, and workers without it remain only as last-resort failover, where their own daemon would pull the model per its policy.
One catalog, many boxes
The gateway refreshes every worker's catalog on the health-check cadence, so the model list on the gateway is the deduplicated union of the whole farm, answered from cache <a href="#ref-2">[2]</a>. A worker whose catalog cannot be fetched is treated as possibly warm rather than unavailable, so routing never degrades below the plain strategy.
Buy hardware per model class
Model-aware routing is always on in gateway mode, so the design lets you buy hardware per model class instead of making every box able to run everything <a href="#ref-1">[1]</a>. Chat runs where the chat models live, embeddings on the CPU box, vision on the accelerator that suits it. Nothing extra is configured: build the farm, install each worker's intended models, and the gateway does placement.
Placement that keeps failover warm
Keep every latency-critical model on at least two workers so failover stays warm rather than triggering a cold load at the worst moment <a href="#ref-2">[2]</a>. The load-balancing strategy still orders within the warm set, so <code>least-connections</code> or a <code>weighted</code> bias toward the big box composes cleanly with warmth.