10. Shipping Changes Safely

General release method

Every change is a hypothesis. Reduce blast radius, define observable guardrails before release, expose a small cohort, compare against a baseline, and keep rollback faster than diagnosis.

Test → Shadow or preview → Canary → Progressive rollout → Full release
                         ↘ pause or rollback at any stage

Version code, APIs, SDKs, schemas, configuration, prompts, models, and migrations. Backward compatibility and rollback must be designed together; a destructive database migration can make an application rollback useless.

Kintsu canary evidence

Agent V2 at 5% traffic
task success  -7%
p95 latency   +40%
cost/request  +18%

Pause and roll back. The canary exists to limit evidence collection, not to justify continuing a bad release.

Compare system types

• SDK: publish a prerelease, run a compatibility matrix, preserve the old API contract, and monitor errors by SDK version.
• Payment workflow: deploy backward-compatible schema first, dual-read or dual-write when needed, and reconcile before removing old fields.
• Kintsu Agent: version prompts, models, tools, and eval datasets; canary by cohort and monitor task quality as well as HTTP health.

Harder decision

What if task success rises 2%, latency rises 25%, and cost rises 35%? There is no universal answer. Compare the change with the product SLO, customer value, margin, and affected cohort.

Lab

Configure a staged V1/V2 rollout in Kintsu Agent. Write explicit continue, pause, and rollback thresholds before viewing results.