Design: Failure modes and recovery

Document failure modes and recovery strategies for Gas Town operations.

## Critical Failure Modes

### 1. Agent Crash Mid-Operation

**Scenario**: Polecat crashes while committing, Witness crashes while verifying

**Detection**:
- Session suddenly gone (tmux check fails)
- State shows 'working' but no session
- Heartbeat stops (for Witness)

**Recovery**:
- Doctor detects via ZombieSessionCheck
- Capture any recoverable state
- Reset agent state to 'idle'
- For Witness: auto-restart via supervisor or manual gt witness start

### 2. Git State Corruption

**Scenario**: Merge conflict, failed rebase, detached HEAD

**Detection**:
- Git commands fail
- Dirty state that won't commit
- Branch diverged from origin

**Recovery**:
- gt doctor reports git health issues
- Manual intervention recommended
- Severe cases: remove clone, re-clone

### 3. Beads Sync Conflict

**Scenario**: Two polecats modify same issue

**Detection**:
- bd sync fails with conflict
- Beads tombstone mechanism handles most cases

**Recovery**:
- Beads has last-write-wins semantics
- bd sync --force in extreme cases
- Issues may need manual dedup

### 4. Tmux Failure

**Scenario**: Tmux server crashes, socket issues

**Detection**:
- All sessions inaccessible
- "no server running" errors

**Recovery**:
- Kill any orphan processes
- tmux kill-server && tmux start-server
- All agent states reset to idle
- Re-spawn active work

### 5. Claude API Issues

**Scenario**: Rate limits, outages, context limits

**Detection**:
- Sessions hang or produce errors
- Repeated failure patterns

**Recovery**:
- Exponential backoff (handled by Claude Code)
- For context limits: session cycling (mail-to-self)
- For outages: wait and retry

### 6. Disk Full

**Scenario**: Clones, logs, or beads fill disk

**Detection**:
- Write operations fail
- git/bd commands error

**Recovery**:
- Clean up logs: rm ~/.gastown/logs/*
- Remove old polecat clones
- gt doctor --fix can clean some cruft

### 7. Network Failure

**Scenario**: Can't reach GitHub, API servers

**Detection**:
- git fetch/push fails
- Claude sessions hang

**Recovery**:
- Work continues locally
- Queue pushes for later
- Sync when connectivity restored

## Recovery Principles

1. **Fail safe**: Prefer stopping over corrupting
2. **State is recoverable**: Git and beads have recovery mechanisms
3. **Doctor heals**: gt doctor --fix handles common issues
4. **Emergency stop**: gt stop --all as last resort
5. **Human escalation**: Some failures need Overseer intervention

## Implementation

- Document each failure mode in architecture.md
- Ensure doctor checks cover detection
- Add recovery hints to error messages
- Log all failures for debugging


Mayor handoff mail template

Add MAYOR_HANDOFF mail template to templates.py.

## Template Function

def mayor_handoff(
    active_swarms: List[SwarmStatus],
    rig_status: Dict[str, RigStatus],
    pending_escalations: List[Escalation],
    in_flight_decisions: List[Decision],
    recent_actions: List[str],
    delegated_work: List[DelegatedItem],
    user_requests: List[str],
    next_steps: List[str],
    warnings: Optional[str] = None,
    session_duration: Optional[str] = None,
) -> Message:
    metadata = {
        'template': 'MAYOR_HANDOFF',
        'timestamp': datetime.utcnow().isoformat(),
        'session_duration': session_duration,
        'active_swarm_count': len(active_swarms),
        'pending_escalation_count': len(pending_escalations),
    }
    # ... format sections ...
    return Message.create(
        sender='mayor/',
        recipient='mayor/',
        subject='Session Handoff',
        body=body,
        priority='high',
    )

## Metadata Fields

- template: MAYOR_HANDOFF
- timestamp: ISO format
- session_duration: Human readable
- active_swarm_count: Number of active swarms
- pending_escalation_count: Number of escalations

## Mail Priority

Use priority='high' to ensure handoff is seen on startup.


Add bulk session stop command (gt session stop --all)

When decommissioning a rig, need to stop multiple sessions one at a time. A --all or --rig flag would allow: gt session stop --rig gastown


Swarm learning: Session cleanup missing from swarm workflow

## Problem

After Enders Game swarm completed (18 issues merged), 16 polecat sessions were left running but idle. No automated cleanup occurred.

## What Should Happen

1. Witness detects polecat completed work (idle at prompt)
2. Witness verifies git state is clean
3. Witness shuts down session
4. Witness reports completion to Mayor

## GGT Components

- gt-cxx: Witness context cycling (covers self-cycling)
- gt-u1j.9: Witness daemon heartbeat loop
- gt-kmn.6: Witness swarm landing protocol

## Recommendation

Add to Witness responsibilities:
- Monitor for 'work complete' signals (DONE keyword, idle detection)
- Automated session shutdown after verification
- Swarm completion reporting to Mayor

See also: architecture.md 'Worker Cleanup (Witness-Owned)' section which describes this but it wasn't implemented in PGT.

Addressed by gt-u1j.9 (daemon heartbeat) which now includes session lifecycle management and worker shutdown. See architecture.md Key Decision #12.


session stop: --force flag is defined but not used


Document merge queue architecture

Update docs/architecture.md with:

- Merge Queue section explaining Beads-native approach
- Engineer role (renamed from Refinery)
- Session restart protocol
- gt mq command reference
- Federation considerations for queue

Also update any references to .gastown/ → config/.


Plugin: Plan-to-Epic converter

## Purpose

Help users create beads epics from various planning inputs.

## Inputs
- Markdown task lists
- GitHub issues
- Linear/Jira exports
- Free-form descriptions
- Existing beads epics

## Output
- Beads epic with properly structured children
- Dependencies set for wave ordering
- Priorities assigned
- Ready for `gt spawn --epic <id>`

## Implementation Options

### Option A: CLI Tool
```bash
gt plan import --from github --repo owner/repo --label batch-candidate
gt plan import --from markdown tasks.md
gt plan structure <epic-id>  # analyze and add dependencies
```

### Option B: Plugin Agent
A plugin at `<rig>/plugins/plan-oracle/` that:
- Receives planning requests via mail
- Analyzes scope and requirements
- Creates structured beads epic
- Sets dependencies based on analysis

### Option C: Interactive Mode
```bash
gt plan create
# Walks through questions, creates epic interactively
```

## Axiom

As stated: 'The Planning phase should end in the creation of a workable Beads plan.'

This plugin bridges the gap between human planning and machine-executable work.

## Priority

P2 - Nice to have for MVP. Manual epic creation works for now.

## Note

No "swarm IDs" - output is just a beads epic with children. Workers process it independently.


gt rig info: Show detailed rig information

Add 'gt rig info <rig>' command to show detailed rig status.

Should show:
- Rig path and git URL
- Active polecats with status
- Refinery status
- Witness status
- Recent activity
- Beads summary (open issues count)

Implemented gt rig info command with detailed rig status display


Swarm learning: Beads database locality gap

## PGT Bug (GGT Design Already Correct)

Mayor created issues in `~/ai/mayor/rigs/beads/.beads/` but polecats use `~/ai/beads/.beads/`. Different databases = polecats can't see Mayor's beads.

**GGT Fix**: architecture.md already specifies:
- All agents use BEADS_DIR pointing to rig-level `.beads/`
- Lines 116-143: Beads Configuration for Multi-Agent
- Lines 573-586: Rig-Level Beads via BEADS_DIR

This is a PGT implementation gap, not a design issue. GGT spawn must set BEADS_DIR correctly.

PGT implementation gap documented. GGT design already correct - architecture.md specifies BEADS_DIR for all agents pointing to rig-level .beads/


Per-rig beads repo configuration

Add per-rig beads configuration to rig config schema.

## Config Schema

In each rig's config.json:

```json
{
  "version": 1,
  "name": "wyvern",
  "git_url": "https://github.com/steveyegge/wyvern",
  "beads": {
    "repo": "local",    // "local" | "<path>" | "<git-url>"
    "root": null,       // Override bd --root (optional)
    "prefix": "wyv"     // Issue prefix for this rig
  }
}
```

## Repo Options

| Value | Meaning | Use Case |
|-------|---------|----------|
| `"local"` | Use project's `.beads/` | Own projects, full commit access |
| `"<path>"` | Use beads at path | OSS contributions |
| `"<git-url>"` | Clone and use repo | Team shared beads |

## Environment Injection

When spawning polecats, Gas Town sets:
```bash
export BEADS_ROOT="<resolved-path>"
```

## Resolution Logic

```go
func ResolveBeadsRoot(rigConfig *RigConfig, rigPath string) (string, error) {
    beads := rigConfig.Beads
    switch {
    case beads.Root != "":
        return beads.Root, nil
    case beads.Repo == "local" || beads.Repo == "":
        return filepath.Join(rigPath, ".beads"), nil
    case strings.HasPrefix(beads.Repo, "/"):
        return beads.Repo, nil
    case strings.Contains(beads.Repo, "://"):
        return cloneAndResolve(beads.Repo)
    default:
        return filepath.Join(rigPath, beads.Repo), nil
    }
}
```

## Backwards Compatibility

If `beads` section missing, assume `"repo": "local"`.