loading…
A customer report is a symptom, not yet a root cause: “It worked last week, now it fails, and nothing changed.” Do not begin by deciding whether the problem belongs to the customer or your team. Establish impact and build a timeline.
Impact → Timeline → Changes → Evidence → Reproduce → Isolate → Mitigate → Escalate
Ask what is affected, when it started, whether it is continuous or intermittent, and what the last known successful request was. Correlate that window with customer changes and your own SDK, API, configuration, certificate, dependency, and infrastructure changes.
Move through the request path one boundary at a time:
Customer application
→ SDK and dependencies
→ Runtime and operating environment
→ Network, proxy, DNS, and TLS
→ Authentication and authorization
→ Public API
→ Internal service
→ Database or provider
Change one variable per experiment. Compare raw HTTP with the SDK, old and new versions, success and failure payloads, and the customer's environment with a clean environment. “Cannot reproduce” is a result that narrows the environment; it is not a resolution.
Ten thousand requests may run at the same time:
09:41:03 get_customer 200
09:41:05 update_customer started
09:41:35 TIMEOUT
09:41:36 retry update_customer
09:41:37 update_customer 409
These lines suggest a timeout followed by a conflicting retry, but they cannot prove which customer, request, deployment, or attempt produced them.
Use structured logs with a request ID and trace ID. Record operation, attempt, duration, dependency, outcome, error category, SDK and runtime version, region, and deployment version. For AI paths, also record model, prompt, tool, and policy versions. Metrics show patterns; traces reconstruct one path; logs explain events.
Never log API keys, credentials, full customer records, or unrestricted prompts.