Auto-draft a Tier-0 runbook from a successful novel (Tier-1) fix

jmz commented

2026-09-10 21:12:43 +00:00

Owner

Goal: close the 'border-collie' learning loop — after the LLM (Tier-1) resolves a novel incident once, warden should PROPOSE a deterministic Tier-0 invoke: runbook so the same issue is fixed at $0/no-LLM next time.

Today: the ladder + memory exist (Tier-0 runbook = the deterministic memory; Forgejo case record + telemetry = the issue log), but a human must hand-author the runbook after seeing the first fix. The self-promotion step is missing.

Proposal: when a Tier-1 attempt resolves an incident using exactly the composed warden invoke <cap> --target ... --param ... tool(s) (M4 composition), capture the successful capability+params and emit a DRAFT runbook (invoke: {capability, params, scope, verify}) attached to the Forgejo case for human review/merge — NOT auto-activated. Optionally a warden runbook propose <incident> CLI.

Guardrails: draft-only + human approval before it becomes an active auto runbook (auto+invoke requires verify at load — keep that); only promote when the fix was a clean single-capability invoke with enumerable params; never auto-promote a free-form shell/edit fix. Record provenance (which incident/session authored it).

Aspirational per the design; this is the concrete first step. Surfaced 2026-09-10.

**Goal:** close the 'border-collie' learning loop — after the LLM (Tier-1) resolves a novel incident once, warden should PROPOSE a deterministic Tier-0 `invoke:` runbook so the same issue is fixed at $0/no-LLM next time. **Today:** the ladder + memory exist (Tier-0 runbook = the deterministic memory; Forgejo case record + telemetry = the issue log), but a human must hand-author the runbook after seeing the first fix. The self-promotion step is missing. **Proposal:** when a Tier-1 attempt resolves an incident using exactly the composed `warden invoke <cap> --target ... --param ...` tool(s) (M4 composition), capture the successful capability+params and emit a DRAFT runbook (invoke: {capability, params, scope, verify}) attached to the Forgejo case for human review/merge — NOT auto-activated. Optionally a `warden runbook propose <incident>` CLI. **Guardrails:** draft-only + human approval before it becomes an active auto runbook (auto+invoke requires verify at load — keep that); only promote when the fix was a clean single-capability invoke with enumerable params; never auto-promote a free-form shell/edit fix. Record provenance (which incident/session authored it). Aspirational per the design; this is the concrete first step. Surfaced 2026-09-10.

jmz commented

2026-09-10 22:30:00 +00:00

Author

Owner

Shipped in v0.20.0. After a Tier-1 (LLM) fix resolves an incident via a mesh invoke, warden attaches a PROPOSED Tier-0 runbook to the Forgejo case: draft-only (auto:false; a human reviews, merges, and sets auto:true to activate), drafted only for a clean single reproducible ok-invoke (free-form/ambiguous fixes never drafted), best-effort + gated (needs Forgejo+mesh; never perturbs the resolve or gate), round-trips through the real loader, provenance recorded, warden_runbook_drafts_total{check_id} metric. Loop confirmed non-inert (exec recorded against incident id). Whole-branch review: APPROVE, no findings.

jmz closed this issue

2026-09-10 22:30:00 +00:00

jmz referenced this issue from a commit

2026-09-10 22:30:01 +00:00

plan: auto-draft Tier-0 runbook from a novel Tier-1 fix (#10)

jmz referenced this issue from a commit

2026-09-10 22:30:01 +00:00

feat(runbook-draft): drafter core + exec_log getter (OSS #10 T1)

jmz referenced this issue from a commit

2026-09-10 22:30:01 +00:00

harden(draft): refuse a capability the runbook loader would reject (#10 T1 review)

jmz referenced this issue from a commit