Deacon Kernel: event processing molecule architecture

Extend the Deacon from "health orchestrator" to "event processing kernel" that runs a
molecule (rounds) each wake cycle, like an OS kernel scheduler.

## Core Insight

The Deacon already has:
- Wake cycle (daemon poke, timer, lifecycle request)
- Mail as event queue
- Health scanning
- Lifecycle processing

But it is REACTIVE. This epic makes it PROACTIVE with a structured molecule:

```
Deacon Wake
├── Step 0: Session Check (should I cycle myself?)
├── Step 1: Health Scan (Mayor, Witnesses, Crew, Refineries)
├── Step 2: Lifecycle Processing (cycle/restart/shutdown requests)
├── Step 3: Plugin Execution (scheduled, event-triggered, human-requested)
├── Step 4: Event Callbacks (timers, subscriptions)
└── Step 5: Heartbeat + Wait
```

## Key Design Decisions

1. **Session cycling first** - Token costs are quadratic with session length.
   Deacon should aggressively cycle to fresh sessions. First step of every
   wake is "do I need a new session?" based on context budget.

2. **Pinned bead with marching orders** - A pinned bead (like gt prime for
   agents) defines the Deacon's rounds molecule. This is the Deacon's
   "kernel" - its operating loop.

3. **Keepalive file for daemon quieting** - Instead of daemon tmux-poking
   the Deacon (which interrupts work), Deacon writes a keepalive file.
   Daemon reads it and backs off. Worst case: Deacon forgets, gets one
   extra heartbeat it squelches.

4. **Molecule-based rounds** - The wake cycle becomes a molecule with steps.
   Each step can have plugins (inline attention). Steps can be added/modified
   by updating the pinned bead.

## Architecture

```
Go Daemon (minimal, dumb)
├── Reads {townRoot}/deacon/keepalive.json
├── If fresh: do nothing
├── If stale: poke Deacon (fallback)
└── Process lifecycle requests (still daemon's job)

Deacon (Claude agent, smart)
├── Wakes: poke, timer, mail event
├── Writes keepalive (quiets daemon for duration of rounds)
├── Runs rounds molecule:
│   ├── session-check: cycle if context budget low
│   ├── health-scan: check agents, remediate
│   ├── lifecycle: process requests
│   ├── plugins: run scheduled/triggered plugins
│   ├── events: process timer/event callbacks
│   └── complete: update heartbeat, wait
└── Reads marching orders from pinned bead
```

## Relation to Existing Work

- Extends gt-5af (Deacon infrastructure) - CLOSED
- Relates to gt-axz (Plugin architecture) - P3
- Fits into phase3-deacon of gt-ngpz (Christmas Plan)
- Enables gt-976 (Crew lifecycle) as part of health-scan step


gt polecat git-state command for pre-kill verification

Add git-state subcommand to gt polecat for Witness pre-kill verification.

## Usage
```bash
gt polecat git-state <rig>/<polecat>
```

## Output
```
Git State: gastown/furiosa

  Working Tree: clean | dirty
  Uncommitted:  0 files | N files (list)
  Unpushed:     0 commits | N commits ahead
  Stashes:      0 | N stashes

  Verdict: CLEAN (safe to kill) | DIRTY (needs cleanup)
```

## JSON output (--json flag)
```json
{
  "clean": true,
  "uncommitted_files": [],
  "unpushed_commits": 0,
  "stash_count": 0
}
```

## Used by
- Witness pre-kill verification (mol-witness-patrol)
- Manual cleanup checks

## Implementation
- Check git status in polecat worktree
- Check git log origin/main..HEAD
- Check git stash list


Merge: gt-h6eq.7

branch: polecat/dag
target: main
source_issue: gt-h6eq.7
rig: gastown


Digest: mol-deacon-patrol

Patrol 9: All healthy


gt crew at/attach: Start session in crew workspace

Implement 'gt crew at <rig>/<name>' and 'gt crew attach <name>' commands:
- Start tmux session (optional - could just print cd instructions)
- Launch claude code in the workspace
- Deliver any pending mail
- Support --no-tmux to just print directory path
- 'attach' infers rig from cwd


Test Polecat Arm

Test child for bonding pattern


Merge: gt-g44u.3

branch: polecat/Ace
target: main
source_issue: gt-g44u.3
rig: gastown


Digest: mol-deacon-patrol

Patrol 13: All healthy


Check Witness and Refinery health

Check Witness and Refinery health for each rig.

**ZFC Principle**: You (Claude) make the judgment call about what is "stuck" or "unresponsive" - there are no hardcoded thresholds in Go. Read the signals, consider context, and decide.

For each rig, run:
```bash
gt witness status <rig>
gt refinery status <rig>
```

**Signals to assess:**

| Component | Healthy Signals | Concerning Signals |
|-----------|-----------------|-------------------|
| Witness | State: running, recent activity | State: not running, no heartbeat |
| Refinery | State: running, queue processing | Queue stuck, merge failures |

**Tracking unresponsive cycles:**

Maintain in your patrol state (persisted across cycles):
```
health_state:
  <rig>:
    witness:
      unresponsive_cycles: 0
      last_seen_healthy: <timestamp>
    refinery:
      unresponsive_cycles: 0
      last_seen_healthy: <timestamp>
```

**Decision matrix** (you decide the thresholds based on context):

| Cycles Unresponsive | Suggested Action |
|---------------------|------------------|
| 1-2 | Note it, check again next cycle |
| 3-4 | Attempt restart: gt witness restart <rig> |
| 5+ | Escalate to Mayor with context |

**Restart commands:**
```bash
gt witness restart <rig>
gt refinery restart <rig>
```

**Escalation:**
```bash
gt mail send mayor/ -s "Health: <rig> <component> unresponsive" \
  -m "Component has been unresponsive for N cycles. Restart attempts failed.
      Last healthy: <timestamp>
      Error signals: <details>"
```

Reset unresponsive_cycles to 0 when component responds normally.


Molecule converge: iterate until AI says done

Work isn't 'done' until AI is confident it's done. Fixes the 85% problem.

**From VC**: internal/iterative/converge.go - 3-7 iterations with AI convergence detection. ~600 lines.
Target: 20%+ more issues discovered. Observed: ~25% average improvement.

**Gas Town implementation**: Molecule converge config:
```yaml
converge:
  strategy: ai
  min_iterations: 3
  max_iterations: 7
  confidence: 0.85
```

Polecat iterates, AI checks convergence. Stop when AI is confident or max reached.

**Value**: Raw agents get to ~85% and stop. Iteration catches the remaining 15%.

**VC metrics**: Convergence rate >70%, mean 4-5 iterations for complex work.


mol-deacon-patrol

Mayor's daemon patrol loop.

The Deacon is the Mayor's background process that runs continuously,
handling callbacks, monitoring rig health, and performing cleanup.
Each patrol cycle runs these steps in sequence, then loops or exits.


mol-deacon-patrol


Day 1.4: Create agent beads for core roles

Create agent beads with deterministic IDs:
- gt-mayor (Mayor coordinator)
- gt-witness (per-rig polecat monitor)
- gt-refinery (per-rig merge processor)

Each with role_type, rig, and initial state.

Parent: gt-d0jqp


Expansion operators: expand and map (macros)

Implement macro-style expansion operators:
- expand(target, template) - apply template to single step
- map(molecule, template) - apply template to all matching steps

Templates use {target} substitution for step references.
Rule of Five is the canonical example.


session-gc

Clean dead sessions. Run gt gc --sessions.


CLI: doctor diagnostics and auto-repair

GGT completely lacks the doctor command which is critical for debugging.

Required Commands:
- gt doctor [<rig>] - Run diagnostic checks
- gt doctor --fix - Auto-repair common issues

Checks to Implement:
Workspace Level: Config validity, Mayor mailbox, Rig registry
Rig Level: Git state, clone health, witness/refinery presence, beads sync
Swarm Level: Stuck detection, zombie sessions, heartbeat health

PGT Reference: gastown-py/src/gastown/cli/dashboard_cmd.py


Digest: mol-deacon-patrol @ 2025-12-24 20:23

Patrol complete: inbox clear, all agents healthy, no orphans


implement

Implement the solution for gt-test123. Follow codebase conventions.
File discovered work as new issues with bd create.

Make regular commits with clear messages.
Keep changes focused on the assigned issue.

Depends: load-context


Digest: mol-deacon-patrol

Patrol 12: nominal


gt sling: use mail queue for patrol roles (witness, refinery, deacon)

When slinging work to patrol agents, queue via mail instead of replacing hook.

**Rationale:**
Patrol agents run continuous loops. Replacing their hook with discrete work breaks patrol continuity - when the task completes, the patrol stops.

**New behavior for patrol roles (witness, refinery, deacon):**
1. Check if patrol is running (hook has patrol molecule attached)
2. If patrol running:
   - Don't touch hook (patrol stays pinned)
   - Send work assignment mail
   - Print 'Queued for next patrol cycle'
3. If patrol NOT running:
   - Start default patrol for that role (mol-witness-patrol, mol-refinery-patrol, mol-deacon-patrol)
   - Send work assignment mail
   - Print 'Started patrol and queued work'

**New flags:**
- --urgent: Interrupt current patrol cycle, process this work immediately
- --replace: Explicitly terminate patrol and pin discrete work (break-glass)

**No change for:**
- Polecat, Crew, Mayor (discrete task agents - current behavior)

**Dependencies:**
- Patrol templates must have 'check inbox' step (verify/add)
- Need to know default patrol molecule for each role


Implement verification patterns for boot steps

Define and implement the verification logic Mayor uses.

## Verification Utilities

Mayor needs to check:

### Session Exists
```bash
tmux has-session -t <session> 2>/dev/null && echo "exists"
```

### Not Stalled (Claude waiting at prompt)
```bash
gt peek <agent> 10 | grep -q "> Try" && echo "stalled"
```

Pattern: `> Try "..."` indicates Claude started but no message sent.

### Recent Heartbeat
```bash
# For deacon
age=$(( $(date +%s) - $(stat -f %m ~/gt/deacon/heartbeat.json) ))
[ $age -lt 120 ] && echo "fresh"
```

### Active Work Output
```bash
gt peek <agent> 20 | grep -q "⏺" && echo "working"
```

Pattern: `⏺` indicates Claude is executing tools.

## Mayor Execution Pattern

```
for each step in molecule:
  run action
  loop:
    check verification
    if verified: close step, continue
    if stalled: run on_stall recovery
    if timeout (5 attempts): log warning, continue
    sleep 10s
```

No hard timeout - Mayor keeps trying with increasing backoff.


Doctor check: Zombie session cleanup

Detect and clean up zombie tmux sessions via gt doctor.

## Problem

Zombie sessions occur when:
- Agent crashes without cleanup
- gt kill fails mid-operation
- System restart leaves orphan sessions
- Session naming collision

## Checks

### ZombieSessionCheck
- List all tmux sessions matching gt-* pattern
- Cross-reference with known polecats
- Flag sessions with no corresponding polecat state
- Flag sessions for removed polecats
- Check session age vs polecat creation time

### Detection Criteria
- Session exists but polecat directory doesn't
- Session name doesn't match any registered polecat
- Polecat state=idle but session running
- Multiple sessions for same polecat

## Output

```
[WARN] Zombie tmux sessions detected:
  - gt-wyvern-OldPolecat (polecat removed)
  - gt-beads-Unknown (no matching polecat)
  - gt-wyvern-Toast (duplicate session)

  Run 'gt doctor --fix' to clean up
```

## Auto-Fix (--fix flag)

- Kill orphan tmux sessions
- Update polecat state to match reality
- Log all cleanup actions

## Safety

- Never kill sessions where polecat state=working
- Prompt before killing if --fix used without --force
- Create audit log of killed sessions


Polecat template: clarify work status commands (bd list vs gt mol status)

Template shows 'bd list --status=in_progress' for checking work, but polecats should probably use 'gt mol status' instead (or in addition). Clarify the right approach for polecats to check their current work assignment.


Digest: mol-deacon-patrol

Patrol 3: all healthy


MR state transitions: validate open→in_progress→closed

Enforce valid state transitions for merge-requests:

Valid transitions:
- open → in_progress (Engineer claims MR)
- in_progress → closed (merge success or rejection)
- in_progress → open (failure, reassign to worker)
- open → closed (manual rejection)

Invalid:
- closed → anything (immutable once closed)

Implement validation in MR update operations.

Reference: docs/merge-queue-design.md#state-machine


Witness-Refinery protocol handlers

Implement handlers for: POLECAT_READY (witness→refinery: worker done, branch ready), MERGE_SUCCESS (refinery→witness: merged, cleanup ok), MERGE_FAILED (refinery→witness: needs rework), REWORK_REQUEST (refinery→witness: rebase needed).


Fanout gate runtime evaluation

Aggregate step uses waits-for: all-children. Deacon must evaluate this gate by checking if all dynamically-bonded arm children are complete. If no children bonded, gate passes immediately.


implement

Implement the solution for {{issue}}. Follow codebase conventions.
File discovered work as new issues with bd create.

Make regular commits with clear messages.
Keep changes focused on the assigned issue.

Depends: load-context


Digest: mol-deacon-patrol

Patrol 19: Quiet


Remove shell respawn loops from witness and deacon

Currently witness and deacon use shell loops (`while true; do claude; done`) to auto-respawn when Claude exits. This bypasses the proper lifecycle architecture.

## Current Behavior
- witness.go: ensureWitnessSession() wraps Claude in a shell loop
- deacon.go: runDeaconStart() wraps Claude in a shell loop
- When Claude exits, shell automatically restarts it

## Desired Behavior
- Agents should request lifecycle changes via LIFECYCLE mail to deacon
- Daemon processes lifecycle requests and handles restarts
- No shell loops - daemon and deacon's health-scan handle respawns

## Why Change
1. Shell loops bypass state verification (requesting_cycle flag)
2. Shell loops bypass handoff mail protocol
3. Shell loops make lifecycle tracking harder
4. Daemon already has infrastructure for this

## Migration
1. Remove shell loop from witness (daemon/deacon handles restart)
2. Remove shell loop from deacon (daemon handles restart via ensureDeaconRunning)
3. Verify daemon's health check properly restarts dead agents
4. Update templates if needed

## Risk
Low - we have gt witness restart and gt deacon restart as fallbacks


mol-deacon-patrol

Mayor's daemon patrol loop.

The Deacon is the Mayor's background process that runs continuously, handling callbacks, monitoring rig health, and performing cleanup. Each patrol cycle runs these steps in sequence, then loops or exits.


Merge: gt-ci84

branch: polecat/slit
target: main
source_issue: gt-ci84
rig: gastown


Pillar 2: Patrol Ignition

Witness and Refinery patrol loops fire reliably.

Key deliverables:
- Witness patrol reads agent beads for polecat state
- Witness sends MERGE_READY to refinery on completion
- Refinery processes MERGE_READY mail
- Both send completion signals

Subsumes existing work:
- gt-6qyt1: Refinery event-driven merge queue
- gt-qpwv4: Witness detect completion
- gt-dtw9u: Witness active monitoring

Reference: ~/gt/docs/liftoff-plan.md

Parent: gt-oki8p


Digest: mol-deacon-patrol

Patrol #19


Digest: mol-deacon-patrol

Patrol 17


mol-deacon-patrol


Digest: mol-deacon-patrol

Patrol 19


Town-level beads: Real DB for coordination mail

Implement Option A from mail redesign: Town gets real beads DB for coordination.

## Background

Mail is now Beads. But currently:
- Town .beads/redirect points to rig beads
- mayor/mail/ has legacy JSONL files
- Cross-rig coordination has no clear home

## Design

Town beads = coordination, cross-rig mail, mayor inbox, handoffs
Rig beads = project issues, work items

Matches HOP hierarchy: platform > project > worker

## Structure

~/gt/
  .beads/           # REAL beads DB (prefix: gm-)
  mayor/
    town.json
    state.json      # NO mail/ directory
  gastown/
    .beads/         # Rig beads (prefix: ga-)

## Tasks

1. Delete ~/gt/.beads/redirect
2. Run bd init --prefix gm at ~/gt/ (town beads)
3. Delete ~/gt/mayor/mail/ directory
4. Update gt mail to use beads not JSONL
5. Add mail fields (thread_id, reply_to, msg_type)
6. Update gt prime for two-tier model
7. Update docs/architecture.md

## Addressing

- mayor/ -> town beads
- rig/agent -> rig beads
- Cross-rig -> town beads


Digest: mol-deacon-patrol

deacon patrol: 21 cycles complete


Build activity stream component

Bubbletea component showing scrollable activity stream for a selected worker. Shows mutation events with timestamps, symbols, and issue titles. Displayed when user expands/selects a worker.


Digest: mol-deacon-patrol

Patrol 13


Digest: mol-deacon-patrol

Patrol 19: Routine


load-context

Run gt prime and bd prime. Verify issue assignment.
Check inbox for any relevant messages.

Read the assigned issue (gt-qwyu) and understand the requirements.
Identify any blockers or missing information.


Verify architecture.md shows correct harness and rig structure

Review architecture.md diagrams:
- Verify town-level structure shows harness correctly
- Confirm rig-level mayor/rig/ is shown (it appears to be there at line 197)
- Check mermaid diagrams match ASCII diagrams
- Update if any inconsistencies found
- Cross-reference with new harness.md docs


Digest: mol-deacon-patrol

Patrol 7: OK


mol-outpost-assign: Intelligent work routing to outposts

The federation design shows outposts.yaml with a static policy section. Simple preference ordering is fine as config, but intelligent work routing should be a molecule.

Current static approach:
```yaml
policy:
  default_preference: [local, gce-burst, cloudrun-burst]
```

Cases requiring cognition:
- "This is a long-running research task, route to VM not CloudRun"
- "This touches sensitive code, keep local"
- "These 5 issues are related, batch them to same outpost"
- "CloudRun cost is high today, prefer local even if slower"

## Molecule: outpost-assign
Intelligent work-to-outpost routing.

## Step: classify-work
Analyze the issue/work item:
- Expected duration (quick fix vs multi-hour)
- Resource requirements
- Sensitivity/security tier
- Related work (same epic?)

## Step: check-capacity
Query outpost status:
- Current load on each
- Cost accrued today
- Health status

## Step: select-outpost
Choose optimal outpost based on:
- Work classification
- Capacity/cost
- Policy constraints
Needs: classify-work, check-capacity

## Step: emit-assignment
Record decision in beads for audit.
Needs: select-outpost

## Notes
- This molecule is invoked by Mayor/Witness when spawning
- Simple cases can short-circuit to static policy
- Full analysis only for ambiguous cases


Day 3.5: Test polecat → refinery → merged flow

Integration test:
1. Spawn polecat with work
2. Polecat completes (updates state, closes bead)
3. Witness detects completion
4. Witness sends MERGE_READY
5. Refinery processes and merges
6. MERGED signal received

Parent: gt-hwka3


Digest: mol-deacon-patrol

Patrol #2: All healthy, no changes


Digest: mol-deacon-patrol

Patrol 13: All green