docs: add plugin and escalation system designs

Plugin System (gt-n08ix): - Deacon-dispatched periodic automation - Dog execution model (non-blocking) - Wisps for state tracking (no state.json) - Gate types: cooldown, cron, condition, event - First plugin: rebuild-gt for stale binary detection Escalation System (gt-i9r20): - Unified gt escalate command with severity routing - Config-driven: settings/escalation.json - Escalation beads for tracking - Stale escalation re-escalation - Actions: bead, mail, email, sms Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 21:24:58 -08:00
parent a106796a0e
commit f5832188a6
2 changed files with 1061 additions and 0 deletions
--- a/docs/design/escalation-system.md
+++ b/docs/design/escalation-system.md
@@ -0,0 +1,576 @@
+# Escalation System Design
+
+> Detailed design for the Gas Town unified escalation system.
+> Written 2026-01-11, crew/george session.
+> Parent epic: gt-i9r20
+
+## Problem Statement
+
+Current escalation is ad-hoc "mail Mayor". Issues:
+- Mayor gets backlogged easily (especially during swarms)
+- No severity differentiation
+- No alternative channels (email, SMS, Slack)
+- No tracking of stale/unacknowledged escalations
+- No visibility into escalation history
+
+## Design Goals
+
+1. **Unified API**: Single `gt escalate` command for all escalation needs
+2. **Severity-based routing**: Different severities go to different channels
+3. **Config-driven**: Town config controls routing, no code changes needed
+4. **Audit trail**: All escalations tracked as beads
+5. **Stale detection**: Unacknowledged escalations re-escalate automatically
+6. **Extensible**: Easy to add new notification channels
+
+---
+
+## Architecture
+
+### Components
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    gt escalate command                       │
+│  --severity --subject --body --source                       │
+└─────────────────────┬───────────────────────────────────────┘
+                      │
+                      ▼
+┌─────────────────────────────────────────────────────────────┐
+│                  Escalation Manager                          │
+│  1. Read config (settings/escalation.json)                  │
+│  2. Create escalation bead                                   │
+│  3. Execute route actions for severity                       │
+└─────────────────────┬───────────────────────────────────────┘
+                      │
+          ┌───────────┼───────────┬───────────┐
+          ▼           ▼           ▼           ▼
+      ┌───────┐  ┌─────────┐  ┌───────┐  ┌───────┐
+      │ Bead  │  │  Mail   │  │ Email │  │  SMS  │
+      │Create │  │ Action  │  │Action │  │Action │
+      └───────┘  └─────────┘  └───────┘  └───────┘
+```
+
+### Data Flow
+
+1. Agent calls `gt escalate --severity=high --subject="..." --body="..."`
+2. Command loads escalation config from `settings/escalation.json`
+3. Creates escalation bead with severity, subject, body, source labels
+4. Looks up route for severity level
+5. Executes each action in the route (bead already created, then mail, email, etc.)
+6. Returns escalation bead ID
+
+### Stale Escalation Flow
+
+1. Deacon patrol (or plugin) runs `gt escalate stale`
+2. Queries for escalation beads older than threshold without `acknowledged:true`
+3. For each stale escalation:
+   - Bump severity (low→medium, medium→high, high→critical)
+   - Re-execute route for new severity
+   - Add `reescalated:true` label and timestamp
+
+---
+
+## Configuration
+
+### File Location
+
+`~/gt/settings/escalation.json`
+
+This follows the existing pattern where `~/gt/settings/` contains town-level behavioral config.
+
+### Schema
+
+```go
+// EscalationConfig represents escalation routing configuration.
+type EscalationConfig struct {
+    Type    string `json:"type"`    // "escalation"
+    Version int    `json:"version"` // schema version
+
+    // Routes maps severity levels to action lists.
+    // Actions are executed in order.
+    Routes map[string][]string `json:"routes"`
+
+    // Contacts contains contact information for actions.
+    Contacts EscalationContacts `json:"contacts"`
+
+    // StaleThreshold is how long before an unacknowledged escalation
+    // is considered stale and gets re-escalated. Default: "4h"
+    StaleThreshold string `json:"stale_threshold,omitempty"`
+
+    // MaxReescalations limits how many times an escalation can be
+    // re-escalated. Default: 2 (low→medium→high, then stops)
+    MaxReescalations int `json:"max_reescalations,omitempty"`
+}
+
+// EscalationContacts contains contact information.
+type EscalationContacts struct {
+    HumanEmail string `json:"human_email,omitempty"`
+    HumanSMS   string `json:"human_sms,omitempty"`
+    SlackWebhook string `json:"slack_webhook,omitempty"`
+}
+
+const CurrentEscalationVersion = 1
+```
+
+### Default Configuration
+
+```json
+{
+  "type": "escalation",
+  "version": 1,
+  "routes": {
+    "low": ["bead"],
+    "medium": ["bead", "mail:mayor"],
+    "high": ["bead", "mail:mayor", "email:human"],
+    "critical": ["bead", "mail:mayor", "email:human", "sms:human"]
+  },
+  "contacts": {
+    "human_email": "",
+    "human_sms": ""
+  },
+  "stale_threshold": "4h",
+  "max_reescalations": 2
+}
+```
+
+### Action Types
+
+| Action | Format | Behavior |
+|--------|--------|----------|
+| `bead` | `bead` | Create escalation bead (always first, implicit) |
+| `mail:<target>` | `mail:mayor` | Send gt mail to target |
+| `email:human` | `email:human` | Send email to `contacts.human_email` |
+| `sms:human` | `sms:human` | Send SMS to `contacts.human_sms` |
+| `slack` | `slack` | Post to `contacts.slack_webhook` |
+| `log` | `log` | Write to escalation log file |
+
+### Severity Levels
+
+| Level | Use Case | Default Route |
+|-------|----------|---------------|
+| `low` | Informational, non-urgent | bead only |
+| `medium` | Needs attention soon | bead + mail mayor |
+| `high` | Urgent, needs human | bead + mail + email |
+| `critical` | Emergency, immediate | bead + mail + email + SMS |
+
+---
+
+## Escalation Beads
+
+### Bead Format
+
+```yaml
+id: gt-esc-abc123
+type: escalation
+status: open
+title: "Plugin FAILED: rebuild-gt"
+labels:
+  - severity:high
+  - source:plugin:rebuild-gt
+  - acknowledged:false
+  - reescalated:false
+  - reescalation_count:0
+description: |
+  Build failed: make returned exit code 2
+
+  ## Context
+  - Source: plugin:rebuild-gt
+  - Original severity: medium
+  - Escalated at: 2026-01-11T19:00:00Z
+created_at: 2026-01-11T15:00:00Z
+```
+
+### Label Schema
+
+| Label | Values | Purpose |
+|-------|--------|---------|
+| `severity:<level>` | low, medium, high, critical | Current severity |
+| `source:<type>:<name>` | plugin:rebuild-gt, patrol:deacon | What triggered it |
+| `acknowledged:<bool>` | true, false | Has human acknowledged |
+| `reescalated:<bool>` | true, false | Has been re-escalated |
+| `reescalation_count:<n>` | 0, 1, 2, ... | Times re-escalated |
+| `original_severity:<level>` | low, medium, high | Initial severity |
+
+---
+
+## Commands
+
+### gt escalate
+
+Create a new escalation.
+
+```bash
+gt escalate \
+  --severity=<low|medium|high|critical> \
+  --subject="Short description" \
+  --body="Detailed explanation" \
+  [--source="plugin:rebuild-gt"]
+```
+
+**Flags:**
+- `--severity` (required): Escalation severity level
+- `--subject` (required): Short description (becomes bead title)
+- `--body` (required): Detailed explanation (becomes bead description)
+- `--source`: Source identifier for tracking (e.g., "plugin:rebuild-gt")
+- `--dry-run`: Show what would happen without executing
+- `--json`: Output escalation bead ID as JSON
+
+**Exit codes:**
+- 0: Success
+- 1: Config error or invalid flags
+- 2: Action failed (e.g., email send failed)
+
+**Example:**
+```bash
+gt escalate \
+  --severity=high \
+  --subject="Plugin FAILED: rebuild-gt" \
+  --body="Build failed: make returned exit code 2. Working directory: ~/gt/gastown/crew/george" \
+  --source="plugin:rebuild-gt"
+
+# Output:
+# ✓ Created escalation gt-esc-abc123 (severity: high)
+# → Created bead
+# → Mailed mayor/
+# → Emailed steve@example.com
+```
+
+### gt escalate ack
+
+Acknowledge an escalation.
+
+```bash
+gt escalate ack <bead-id> [--note="Investigating"]
+```
+
+**Behavior:**
+- Sets `acknowledged:true` label
+- Optionally adds note to bead
+- Prevents re-escalation
+
+**Example:**
+```bash
+gt escalate ack gt-esc-abc123 --note="Looking into it"
+# ✓ Acknowledged gt-esc-abc123
+```
+
+### gt escalate list
+
+List escalations.
+
+```bash
+gt escalate list [--severity=...] [--stale] [--unacked] [--all]
+```
+
+**Flags:**
+- `--severity`: Filter by severity level
+- `--stale`: Show only stale (past threshold, unacked)
+- `--unacked`: Show only unacknowledged
+- `--all`: Include acknowledged/closed
+- `--json`: Output as JSON
+
+**Example:**
+```bash
+gt escalate list --unacked
+# 📢 Unacknowledged Escalations (2)
+#
+#   ● gt-esc-abc123 [HIGH] Plugin FAILED: rebuild-gt
+#     Source: plugin:rebuild-gt · Age: 2h · Stale in: 2h
+#   ● gt-esc-def456 [MEDIUM] Witness unresponsive
+#     Source: patrol:deacon · Age: 30m · Stale in: 3h30m
+```
+
+### gt escalate stale
+
+Check for and re-escalate stale escalations.
+
+```bash
+gt escalate stale [--dry-run]
+```
+
+**Behavior:**
+- Queries unacked escalations older than `stale_threshold`
+- For each, bumps severity and re-executes route
+- Respects `max_reescalations` limit
+
+**Example:**
+```bash
+gt escalate stale
+# 🔄 Re-escalating stale escalations...
+#
+#   gt-esc-abc123: medium → high (age: 5h, reescalation: 1/2)
+#   → Emailed steve@example.com
+#
+# ✓ Re-escalated 1 escalation
+```
+
+### gt escalate close
+
+Close an escalation (resolved).
+
+```bash
+gt escalate close <bead-id> [--reason="Fixed in commit abc123"]
+```
+
+**Behavior:**
+- Sets status to closed
+- Adds resolution note
+- Records who closed it
+
+---
+
+## Implementation Details
+
+### File: internal/cmd/escalate.go
+
+```go
+package cmd
+
+// escalateCmd is the parent command for escalation management.
+var escalateCmd = &cobra.Command{
+    Use:   "escalate",
+    Short: "Manage escalations",
+    Long:  `Create, acknowledge, and manage escalations with severity-based routing.`,
+}
+
+// escalateCreateCmd creates a new escalation.
+var escalateCreateCmd = &cobra.Command{
+    Use:   "escalate --severity=<level> --subject=<text> --body=<text>",
+    Short: "Create a new escalation",
+    // ... implementation
+}
+
+// escalateAckCmd acknowledges an escalation.
+var escalateAckCmd = &cobra.Command{
+    Use:   "ack <bead-id>",
+    Short: "Acknowledge an escalation",
+    // ... implementation
+}
+
+// escalateListCmd lists escalations.
+var escalateListCmd = &cobra.Command{
+    Use:   "list",
+    Short: "List escalations",
+    // ... implementation
+}
+
+// escalateStaleCmd checks for stale escalations.
+var escalateStaleCmd = &cobra.Command{
+    Use:   "stale",
+    Short: "Re-escalate stale escalations",
+    // ... implementation
+}
+
+// escalateCloseCmd closes an escalation.
+var escalateCloseCmd = &cobra.Command{
+    Use:   "close <bead-id>",
+    Short: "Close an escalation",
+    // ... implementation
+}
+```
+
+### File: internal/escalation/manager.go
+
+```go
+package escalation
+
+// Manager handles escalation creation and routing.
+type Manager struct {
+    config *config.EscalationConfig
+    beads  *beads.Client
+    mailer *mail.Client
+}
+
+// Escalate creates a new escalation and executes the route.
+func (m *Manager) Escalate(ctx context.Context, opts EscalateOptions) (*Escalation, error) {
+    // 1. Validate options
+    // 2. Create escalation bead
+    // 3. Look up route for severity
+    // 4. Execute each action
+    // 5. Return escalation with results
+}
+
+// Acknowledge marks an escalation as acknowledged.
+func (m *Manager) Acknowledge(ctx context.Context, beadID string, note string) error {
+    // 1. Load escalation bead
+    // 2. Set acknowledged:true label
+    // 3. Add note if provided
+}
+
+// ReescalateStale finds and re-escalates stale escalations.
+func (m *Manager) ReescalateStale(ctx context.Context) ([]Reescalation, error) {
+    // 1. Query unacked escalations older than threshold
+    // 2. For each, bump severity
+    // 3. Execute new route
+    // 4. Update labels
+}
+```
+
+### File: internal/escalation/actions.go
+
+```go
+package escalation
+
+// Action is an escalation route action.
+type Action interface {
+    Execute(ctx context.Context, esc *Escalation) error
+    String() string
+}
+
+// BeadAction creates the escalation bead.
+type BeadAction struct{}
+
+// MailAction sends gt mail.
+type MailAction struct {
+    Target string // e.g., "mayor"
+}
+
+// EmailAction sends email.
+type EmailAction struct {
+    Recipient string // from config.contacts
+}
+
+// SMSAction sends SMS.
+type SMSAction struct {
+    Recipient string // from config.contacts
+}
+
+// ParseAction parses an action string into an Action.
+func ParseAction(s string) (Action, error) {
+    // "bead" -> BeadAction{}
+    // "mail:mayor" -> MailAction{Target: "mayor"}
+    // "email:human" -> EmailAction{Recipient: "human"}
+    // etc.
+}
+```
+
+### Email/SMS Implementation
+
+For v1, use simple exec of external commands:
+
+```go
+// EmailAction sends email using the 'mail' command or similar.
+func (a *EmailAction) Execute(ctx context.Context, esc *Escalation) error {
+    // Option 1: Use system mail command
+    // Option 2: Use sendgrid/ses API (future)
+    // Option 3: Use configured webhook
+
+    // For now, just log a placeholder
+    // Real implementation can be added based on user's infrastructure
+}
+```
+
+The email/SMS actions can start as stubs that log warnings, with real implementations added based on the user's infrastructure (SendGrid, Twilio, etc.).
+
+---
+
+## Integration Points
+
+### Plugin System
+
+Plugins use escalation for failure notification:
+
+```markdown
+# In plugin.md execution section:
+
+On failure:
+```bash
+gt escalate \
+  --severity=medium \
+  --subject="Plugin FAILED: rebuild-gt" \
+  --body="$ERROR" \
+  --source="plugin:rebuild-gt"
+```
+```
+
+### Deacon Patrol
+
+Deacon uses escalation for health issues:
+
+```bash
+# In health-scan step:
+if [ $unresponsive_cycles -ge 5 ]; then
+  gt escalate \
+    --severity=high \
+    --subject="Witness unresponsive: gastown" \
+    --body="Witness has been unresponsive for $unresponsive_cycles cycles" \
+    --source="patrol:deacon:health-scan"
+fi
+```
+
+### Stale Escalation Check
+
+Can be either:
+1. A Deacon patrol step
+2. A plugin (dogfood!)
+3. Part of `gt escalate` itself (run periodically)
+
+Recommendation: Start as patrol step, migrate to plugin later.
+
+---
+
+## Testing Plan
+
+### Unit Tests
+
+- Config loading and validation
+- Action parsing
+- Severity level ordering
+- Re-escalation logic
+
+### Integration Tests
+
+- Create escalation → bead exists
+- Acknowledge → label updated
+- Stale detection → re-escalation triggers
+- Route execution → all actions called
+
+### Manual Testing
+
+1. `gt escalate --severity=low --subject="Test" --body="Testing"`
+2. `gt escalate list --unacked`
+3. `gt escalate ack <id>`
+4. Wait for stale threshold, run `gt escalate stale`
+
+---
+
+## Dependencies
+
+### Internal Dependencies (task order)
+
+```
+gt-i9r20.2 (Config Schema)
+    │
+    ▼
+gt-i9r20.1 (gt escalate command)
+    │
+    ├──▶ gt-i9r20.4 (gt escalate ack)
+    │
+    └──▶ gt-i9r20.3 (Stale patrol)
+```
+
+### External Dependencies
+
+- `bd create` for creating escalation beads
+- `bd list` for querying escalations
+- `bd label` for updating labels
+- `gt mail send` for mail action
+
+---
+
+## Open Questions (Resolved)
+
+1. **Where to store config?** → `settings/escalation.json` (follows existing pattern)
+2. **How to implement email/SMS?** → Start with stubs, add real impl based on infrastructure
+3. **Stale check: patrol step or plugin?** → Start as patrol step, can migrate to plugin
+4. **Escalation bead type?** → `type: escalation` (new bead type)
+
+---
+
+## Future Enhancements
+
+1. **Slack integration**: Post to Slack channels
+2. **PagerDuty integration**: Create incidents
+3. **Escalation dashboard**: Web UI for escalation management
+4. **Scheduled escalations**: "Remind me in 2h if not resolved"
+5. **Escalation templates**: Pre-defined escalation types