loading…
Why software architecture changes when part of the system reasons probabilistically at runtime
Traditional software executes logic that developers explicitly encode. A GenAI system still contains ordinary code, databases, APIs, and infrastructure, but it also contains a model whose response is generated at runtime from instructions and context. That one change introduces a new architectural problem: the same input may not always produce the same output, and a fluent output may still be wrong.
In a conventional application, a function such as calculateTax(order) should return the same result whenever its inputs and configuration are the same. Engineers can inspect the source code, enumerate branches, and write assertions for exact outputs. The application may still fail because of bugs, bad data, or infrastructure problems, but the intended control flow is explicit.
A model call behaves differently. The application supplies instructions and context, and the model produces a likely continuation. Small changes in wording, retrieved documents, conversation history, model version, or sampling settings can change the result. This does not make GenAI systems untestable. It means the system must be designed around acceptable behavior across a range of cases, rather than one exact string.
| Traditional software concern | Additional GenAI concern |
|---|---|
| Is the algorithm implemented correctly? | Does the model behave acceptably across realistic and adversarial inputs? |
| Is the API contract valid? | Does generated output conform to the contract before the application trusts it? |
| Did the request succeed? | Was the answer correct, grounded, safe, and worth its latency and cost? |
| Which code path ran? | Which context, tools, model version, and intermediate decisions shaped the result? |
In GenAI applications, natural-language instructions are no longer merely user-facing copy. They influence system behavior. A system prompt can define a role, a tool description can determine whether an action is selected, and an example can shape the structure of an answer. Prompts therefore deserve many of the same practices as code: ownership, review, versioning, testing, and rollback.
A clever sentence is rarely enough for a production task. The model needs the right information at the right moment. Context engineering is the design of the runtime information environment around the model: what is included, how it is selected, how it is structured, and what is deliberately excluded.
Your original six-part taxonomy remains useful:
| Context type | Architectural purpose | Typical example |
|---|---|---|
| Instructions | Define objectives, boundaries, priorities, and output expectations | "Use only approved customer data and return valid JSON." |
| Knowledge | Supply facts the model cannot reliably know from training alone | Product documentation, policies, schemas, current inventory |
| Memory and state | Preserve relevant progress across turns or tasks | Current plan, completed steps, customer preferences |
| Examples | Demonstrate desired patterns more precisely than rules alone | Approved support responses or code patterns |
| Tools | Describe actions the system can request | Search, database lookup, ticket creation |
| Guardrails | Constrain what the system may read, generate, or execute | Permission checks, spend limits, output validation |
These categories do not all have to be pasted into one giant prompt. Some belong in application code, policy engines, retrieval systems, or tool runtimes. Chapter 2 will separate these building blocks; Chapter 3 will show how different architectures assemble them.
When AI can generate implementation quickly, typing code becomes less dominant as a bottleneck. The difficult work moves outward: defining the problem, supplying context, choosing architecture, constraining actions, and verifying behavior. Requirements become executable hypotheses. A prototype can clarify what users actually need, but it does not remove the need for a specification.
The new SDLC is therefore not "describe anything and let the model build it." It is a tighter loop between intent, implementation, observation, and refinement. The model accelerates production; engineering discipline determines whether the result is dependable.