Guarded Incident-response Agent

Constraints

• The agent is isolated to the agentic-ops-lab namespace.
• Diagnosis is read-only; mutation requires an exact, expiring approval token.
• No shell, exec, secret reads, arbitrary manifests, raw Kubernetes API proxy, or production namespace access.
• The only mutations are restart and bounded scale on one labelled synthetic deployment.

Architecture

A LangGraph diagnosis service calls a separate typed tool broker. The agent service account has no Kubernetes credential. Narrow Kubernetes permissions belong only to the broker service account. A deterministic policy selects scope, cause, and allowed action; an optional Bedrock or OpenRouter model may rewrite the bounded summary but cannot select tools, resources, or actions.

Security and failure modes

The corpus covers approval bypass, prompt injection inside logs, cross-namespace reads, unsafe action arguments, stale evidence, diagnosis uncertainty, and partial remediation failure. The live namespace adds Pod Security restricted enforcement, default-deny networking, quota, encrypted secrets, separate service accounts, and no ingress.

Operations and evaluation

The committed evaluator contains 60 synthetic runs: 12 each for crash loops, sustained latency, bad configuration, resource saturation, and adversarial evidence. Every run attempts a cross-namespace read; every actionable run first attempts mutation without approval and then with an exact approval.

Results

Deterministic policy baseline, 4 August 2026: 60/60 cases passed, with 100% diagnosis match, 100% safe-action match, zero approval bypasses, zero cross-namespace reads, and zero leaked adversarial markers. All 48 approved synthetic actions reached their simulated recovery assertion. Median harness latency was 0.502 ms and p95 was 0.616 ms; model cost was $0 because no model was called.

Separate live-lab validation: Argo CD reported the isolated application Synced and Healthy with all three deployments ready. The read-only live check inspected the healthy demo, selected no action because evidence did not justify one, and safely rejected a cross-namespace request. Kubernetes authorization checks confirmed that the agent cannot read pods or patch deployments; the broker can read lab evidence and patch only incident-demo, but cannot read secrets, use exec, patch another deployment, or read another namespace.

The synthetic recovery figures are not live recovery time, and the live check is not a production incident or autonomous production remediation claim.

Trade-offs

The broker deliberately limits flexibility. It cannot solve every incident, but it makes the allowed evidence and action surface inspectable, testable, revocable, and small enough to reason about.