loading…
Separate request execution from the control systems that configure, evaluate, and govern it.
A production architecture should make model access, data movement, policy, state, and observability visible. One useful distinction is between the data plane, which executes user requests, and the control plane, which governs how that execution is configured and released.
Client → API / identity → Application orchestrator
→ Model gateway → Model
↘ Retrieval / tools / state
↘ Policy checks and telemetry
The data plane carries prompts, retrieved context, tool requests, state transitions, and responses. Sensitive data should cross only the boundaries required for the task.
The control plane manages model and prompt versions, routing rules, policies, evaluation datasets, release approvals, budgets, feature flags, and rollback. It changes how the data plane behaves without embedding every choice in product code.
| Boundary | Purpose |
|---|---|
| Model gateway | Provider abstraction, routing, policy, quotas, and telemetry |
| Tool runtime | Authorization, validation, idempotency, and sandboxing |
| Retrieval service | Ingestion, indexing, access filtering, and citation provenance |
| State service | Sessions, checkpoints, memory lifecycle, and tenant isolation |
| Evaluation pipeline | Regression evidence and release gates |
Can you change a model, prompt, policy, or routing rule safely—and identify exactly which requests used the old and new versions?