Shifting to Context Engineering for Reliable LLM Root Cause Analysis

Most incident teams still ask “can AI actually do RCA?” when they should be asking “what did we let the model see?” [1][2] In modern observability stacks, LLM reasoning is rarely the bottleneck; the constraint is the pipeline that decides which metrics, traces, logs, and topology data enter the context window. [1][3]

💡 Key takeaway: For RCA, you are engineering a constrained operating system for the model’s attention, not just writing a clever prompt. [6][7]

1. Why LLM Root Cause Analysis Is Now a Context Problem

Coroot’s Chaos Mesh experiment makes this shift concrete. Engineer Nikolay Sivko fed a fixed, deterministic ~9,800‑token context from a network delay incident to eleven models. [1][3]

• Frontier models (Claude Opus 4.8, GPT‑5.5, Gemini 3.1 Pro) correctly identified root cause and fix.
• Among self‑hosted models, only Gemma 4 31B matched them, despite misleading but realistic telemetry. [1][3]

This setup separated:

• Reasoning: Given the right data, can the model infer cause → effect → remediation?
• Harness: Does the system select and shape the right data in the first place? [1][3]

Findings:

• With carefully curated context, RCA reasoning is “mostly solved” for well‑instrumented incidents in top‑tier models. [1][2]
• The main challenge is building a harness that reliably surfaces the right evidence.

When an incident agent misdiagnoses, you often can’t tell whether:

• The model’s reasoning failed, or
• The harness hid the key metric spike, topology change, or log event. [2][3]

Agentic RCA systems deepen this ambiguity by:

• Issuing open‑ended queries across logs, metrics, traces.
• Following long tool‑use chains that are hard to replay. [2][3]

Deterministic pipelines like Coroot and Dynatrace Davis AI instead: [1][2][3]

• Pre‑correlate topology and anomalies.
• Produce a compact snapshot.
• Pass that snapshot into a narrow, reproducible LLM step.

These topology‑aware designs win in production because they are:

• Easier to debug (you can replay the exact context).
• Easier to govern (bounded cost, constrained behavior).
• Clearer for attributing failure (model vs. missing evidence). [1][3]

⚠️ Key point: If you can’t reconstruct the exact context that produced a bad RCA, you can’t reliably improve the system—regardless of model quality. [1][6]

2. What Context Engineering Means for LLM-Based RCA

Context engineering in RCA is building dynamic systems that assemble the right telemetry, topology, and incident metadata in the right format so the LLM can infer root cause without guessing around gaps. [4][6]

Compared to classic prompt engineering, which tunes wording, context engineering optimizes the full token budget—instructions, logs, metrics, traces, dependencies—within a finite context window. [5][6][7]

The question shifts from:

• “What is the best prompt?” to
• “What is the best token budget allocation for reliable RCA?” [6][7]

An RCA context window typically combines: [5][6]

• System prompts: SRE practices, safety rules, triage priorities.
• Longer‑term memory: Historical incidents, known failure patterns.
• Artifacts: Topology graphs, runbooks, change tickets.
• Incident buffer: Live alerts, chat, previous tool calls.
• Tool schemas: Contracts for querying observability backends.

💼 Practical failure mode: A support agent with elaborate prompts still “forgot” multi‑step troubleshooting because history and tools never entered context. [8]

Incident agents fail similarly when context is weak: they degrade into short‑term chatbots that:

• Lose workflow state.
• Miss topology and history.
• Misattribute causes. [4][8]

💡 Key takeaway: For observability teams, context engineering is now the core of AI engineering for production incident tooling. [4][6][7]

3. Designing a Context Pipeline for Reliable LLM RCA

A robust RCA pipeline starts before any model call and runs through three stages.

1. Deterministic correlation layer
   Compress the telemetry firehose into a high‑signal snapshot: [1][2]

   • Dependency graph and blast radius.
   • Anomalous metrics and relevant traces.
   • Only logs that changed near incident onset.

   Coroot correlates signals into “findings” before handing them to the model. [1][3]

2. Context selection and layout
   Define explicit policies for:

   • Which services (impacted plus key upstream/downstream).
   • Which time windows (e.g., T‑5 to T+10 minutes).
   • Which metric dimensions and log subsets.

   Prioritize tokens with causal evidence—topology deltas, error bursts, config or deploy changes—over verbose stack traces. [6][7]

   Structure the final prompt like an OS: [5][6][7]

   • Kernel‑level instructions: Objectives, RCA procedure, output schema.
   • Environment knowledge: Topology, SLOs, known failure modes.
   • Current artifacts: Alerts, graphs, correlated findings, changes.
   • Tool schemas: For any follow‑up queries.

   ⚠️ Key point: Token budget is a hard constraint; every low‑value log line displaces a potentially critical clue. [6]

3. Evaluation and observability loop
   Treat the RCA pipeline as an observable system. Track: [1][6]

   • Which signals and services entered context.
   • Context size and composition.
   • Model conclusions, confidence, and downstream actions.
   • Human disagreements and their rationale.

   Reviewing misdiagnoses with full context logs lets teams refine selection rules instead of merely swapping models, yielding more stable RCA over time. [1][6]

Conclusion: Audit Your Context Before You Upgrade Your Model

The center of gravity in LLM‑driven RCA has shifted from “can the model reason?” to “did we engineer the right context?” [1][2] Deterministic, context‑rich pipelines make RCA outcomes more debuggable, auditable, and production‑ready than unconstrained agent loops. [1][3][6]

💡 Next step: Audit current incident workflows. Map what data actually reaches the LLM, identify noisy or missing context, then pilot a small deterministic context‑engineering layer—using Coroot‑style fixed‑context experiments—before scaling across your observability stack. [1][3][6]