8. Operating a Nondeterministic Agent

Evidence

Expected:

{"action":"refund","amount":20}

Observed:

{"decision":"give money back","amount":"$20"}

The same request can also select a different tool. Retrying may repair a malformed sample, but it does not repair a weak contract or ambiguous policy.

Production controls

Require structured output, validate the schema, validate tool arguments, constrain high-risk tools, and route uncertain or unsafe cases to a human. Track repair attempts and failures as product signals.

Lab

Run 50 cases five times in Kintsu Agent. Measure schema-valid rate, task success, tool-selection accuracy, and run-to-run variance. Add validation, bounded repair, fallback, and escalation; then compare the distributions.

Decision rule

Retry when sampling noise can plausibly produce a valid answer. Do not retry when the requested action is forbidden, the tool contract is wrong, or the necessary data is absent.