Centralize model access while keeping domain logic independent of one provider.
◷ Estimated reading time: 5 min
A model gateway is an application-controlled boundary between product services and one or more model runtimes. It should simplify governance and portability—not hide meaningful capability differences behind a lowest-common-denominator API.
Core Responsibilities
Authentication, quotas, and rate-limit handling
Model selection and task-aware routing
Request normalization and provider adapters
Prompt and configuration version metadata
Usage, latency, error, and cost telemetry
Fallback and circuit-breaking policies
Route by Task and Risk
Use simpler models for bounded extraction or classification when evaluation shows they meet the threshold. Reserve more capable models for ambiguous planning or difficult synthesis. Routing must be evaluated end to end: a cheap model that creates retries, escalations, or incorrect actions can raise total cost.
Fallback Is Not Automatic Equivalence
Providers and model versions differ in tool behavior, context limits, output formats, latency, and safety behavior. A fallback must be tested against the same acceptance criteria and may require a reduced-capability mode rather than transparent substitution.
Anti-pattern
Do not let every product team call providers directly with its own retry logic, logging format, and secret management. That creates policy drift and makes incidents difficult to contain.
Key Takeaways
A model gateway centralizes access, policy, routing, and telemetry.
Route according to measured task performance, risk, latency, and total cost.
Fallback models require compatibility and quality testing; they are not automatically interchangeable.
A smaller model costs less per call but doubles retries. Which metric should drive the routing decision?
What is the strongest reason to centralize model access?