loading…
Why production behavior depends on the system around the model
A capable model is necessary, but it is not a complete application. The same model can look excellent in one product and unreliable in another because the surrounding system determines what the model sees, what it may do, how its output is checked, and what happens when it fails. That surrounding system is the harness.
This is not a literal percentage formula. It is a debugging and architecture lens. When performance is poor, the model may be the limiting factor—but so may missing context, unclear tool contracts, stale state, weak retrieval, or absent validation. Upgrading the model before investigating the harness can be expensive and ineffective.
| Harness component | Responsibility | Failure when weak |
|---|---|---|
| Context assembly | Selects instructions, knowledge, examples, history, and current state | The model guesses, misses constraints, or attends to irrelevant information |
| Tool runtime | Defines tools, validates arguments, executes actions, and returns observations | Wrong tools are selected or unsafe side effects occur |
| State management | Tracks progress, checkpoints, artifacts, and session identity | The system repeats work, loses goals, or mixes user data |
| Orchestration | Controls loops, routing, delegation, budgets, and stop conditions | Runaway loops, unnecessary calls, or incomplete tasks |
| Policy and guardrails | Enforces permissions, data boundaries, approvals, and output constraints | Excessive agency, leakage, or policy violations |
| Evaluation | Measures task success, quality, safety, and regressions | The team optimizes anecdotes instead of evidence |
| Observability | Records traces, model versions, tool calls, latency, errors, and cost | Failures cannot be reproduced or diagnosed |
| Execution environment | Provides isolation, credentials, networking, and resource limits | A mistake reaches sensitive systems or consumes uncontrolled resources |
The difference between generating a suggestion and executing an action is architectural. A coding assistant that proposes a shell command is less risky than one that executes it with production credentials. Sandboxes let an agent inspect files, run code, and observe results inside an isolated environment. However, isolation is not a substitute for permissions: an environment can be sandboxed and still expose secrets or outbound network access.
When a result is wrong, classify the failure before changing prompts or switching models:
This ordering prevents "prompt patching," where one local failure is addressed by adding more instructions until the prompt becomes contradictory and brittle.
A well-designed harness separates business behavior from a single model provider. It can route tasks to different models, validate outputs through stable schemas, and preserve observability across upgrades. Complete vendor independence is rarely realistic because model capabilities and APIs differ, but architectural separation reduces migration cost and makes comparison possible.