loading…
Measure the failure modes introduced by each architecture pattern.
There is no universal "AI score." Evaluation should follow the architecture selected in Chapter 3 and isolate the layer that failed.
| Architecture | What to evaluate |
|---|---|
| Direct LLM / chatbot | Instruction following, answer quality, conversation consistency, safety |
| RAG | Retrieval recall, context relevance, groundedness, citation correctness |
| Deterministic workflow | Routing, state transitions, validation, retry and compensation behavior |
| Agentic workflow | Plan quality, tool selection, arguments, trajectory, stopping, cost per success |
| Multi-agent system | Delegation, handoffs, role boundaries, shared-state consistency, coordination overhead |
If a RAG answer is wrong, first ask whether the correct evidence was retrieved. A generator cannot faithfully use evidence it never received. Evaluate retrieval independently with labeled relevant documents, then evaluate whether the answer is supported by the supplied context and whether its citations point to the right claims.
For an agent, the final answer may hide an unsafe or wasteful path. Record tool calls, policy decisions, retries, state changes, and stop reasons. Useful metrics include successful completion rate, unauthorized-action rate, unnecessary steps, loop frequency, human-escalation rate, and cost per successful task.
Multi-agent systems may improve specialization but add handoff failures, duplicated work, latency, and token use. Compare them with a simpler single-agent or workflow baseline. Complexity is justified only when the system-level outcome improves enough to pay for the coordination overhead.