loading…
HTTP success is 99.9%, p95 latency is normal, and infrastructure errors are flat. Task success has fallen to 72%.
Traditional monitoring proves the service responds. Evals test whether the response does the job.
Maintain a versioned golden dataset containing normal cases, boundaries, regressions, and safety-critical cases. Score task success, tool-call accuracy, policy compliance, cost, and latency. Run offline evals before rollout and sample privacy-safe online feedback after rollout.
Use the evals/ suite in Kintsu Agent to compare V1 and V2. If V1 passes 47/50 and V2 passes 43/50, inspect which four regressed before averaging the result away.
A single aggregate threshold is insufficient for high-risk failures. Define required pass rates for critical slices and report quality, latency, and cost together.