beads/COMPACTION_SUMMARY.md

Issue Compaction - Quick Reference

Status: Design Complete, Ready for Implementation Target: Beads v1.1 Estimated Effort: 40-60 hours (MVP)

What Is This?

Add intelligent database compaction to beads using Claude Haiku to semantically compress old, closed issues. Keep your database lightweight while preserving the meaning of past work.

Why?

• Agent Efficiency: Smaller DBs → faster queries → clearer thinking
• Context Management: Agents need summaries, not verbose details
• Pragmatic: Most work is throwaway; forensic value decays exponentially
• Git-Friendly: Smaller JSONL exports

How It Works

Two-Tier Compression

Tier 1: Standard (30 days)

• Closed ≥30 days, all dependents closed (2 levels deep)
• Haiku summarizes to ~300 words
• Keeps: title, summary with key decisions
• Clears: verbose design/notes/criteria
• Result: 70-85% space reduction

Tier 2: Aggressive (90 days)

• Already Tier 1 compressed
• Closed ≥90 days, deep dependencies closed (5 levels)
• ≥100 git commits since closure (measures "project time")
• Ultra-compress to single ≤150 word paragraph
• Optionally prunes old events
• Result: 90-95% space reduction

Safety First

• Snapshots: Full original content saved before compaction
• Restore: bd compact --restore <id> undoes compaction
• Dry-Run: bd compact --dry-run previews without changing anything
• Git Backup: JSONL commits preserve full history
• Opt-In: Disabled by default (auto_compact_enabled = false)

CLI

# Preview candidates
bd compact --dry-run

# Compact Tier 1
bd compact --all

# Compact Tier 2
bd compact --tier 2 --all

# Compact specific issue
bd compact --id bd-42

# Restore compacted issue
bd compact --restore bd-42

# Show statistics
bd compact --stats

Cost

Haiku Pricing:

• ~$0.0008 per issue (Tier 1)
• ~$0.0003 per issue (Tier 2)
• ~$1.10 per 1,000 issues total

Negligible for typical usage (~$0.05/month for active project).

Example Output

Before Compaction

bd-42: Fix authentication bug [CLOSED]

Description: (800 words about the problem)

Design: (1,200 words of implementation notes)

Notes: (500 words of testing/deployment details)

Acceptance Criteria: (300 words of requirements)

Total: 2,341 bytes

After Tier 1 Compaction

bd-42: Fix authentication bug [CLOSED] 🗜️

**Summary:** Fixed race condition in JWT token refresh logic causing
intermittent 401 errors. Implemented mutex-based locking. All users
can now stay authenticated reliably.

**Key Decisions:**
- Used sync.RWMutex for simpler reasoning
- Added exponential backoff to prevent thundering herd
- Preserved token format for backward compatibility

**Resolution:** Deployed Aug 31, zero 401s after 2 weeks monitoring.

---
💾 Restore: bd compact --restore bd-42
📊 Original: 2,341 bytes | Compressed: 468 bytes (80% reduction)

Total: 468 bytes (80% reduction)

Implementation Plan

Phase 1: Foundation (16 hours)

1. Schema changes (compaction columns, snapshots table)
2. Config keys
3. Candidate identification queries
4. Migration testing

Phase 2: Core Compaction (20 hours)

5. Haiku client with prompts
6. Snapshot creation/restoration
7. Tier 1 compaction logic
8. CLI command (bd compact)

Phase 3: Advanced Features (12 hours)

9. Tier 2 compaction
10. Restore functionality
11. Stats command
12. Event tracking

Phase 4: Polish (12 hours)

13. Comprehensive tests
14. Documentation (README, COMPACTION.md)
15. Examples and workflows

Total MVP: ~60 hours

Optional Enhancements (Phase 5+)

• Auto-compaction triggers
• Git commit counting
• Local model support (Ollama)
• Custom prompt templates

Architecture

Issue (closed 45 days ago)
    ↓
Eligibility Check
    ↓ (passes: all deps closed, >30 days)
Create Snapshot
    ↓
Call Haiku API
    ↓ (returns: structured summary)
Update Issue
    ↓
Record Event
    ↓
Export to JSONL

Key Files

New Files:

• internal/compact/haiku.go - Haiku API client
• internal/compact/compactor.go - Core compaction logic
• internal/storage/sqlite/compact.go - Candidate queries
• cmd/bd/compact.go - CLI command

Modified Files:

• internal/storage/sqlite/schema.go - Add snapshots table
• internal/storage/sqlite/sqlite.go - Add migrations
• internal/types/types.go - Add EventCompacted
• cmd/bd/show.go - Display compaction status

Schema Changes

-- Add to issues table
ALTER TABLE issues ADD COLUMN compaction_level INTEGER DEFAULT 0;
ALTER TABLE issues ADD COLUMN compacted_at DATETIME;
ALTER TABLE issues ADD COLUMN original_size INTEGER;

-- New table
CREATE TABLE issue_snapshots (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    issue_id TEXT NOT NULL,
    snapshot_time DATETIME NOT NULL,
    compaction_level INTEGER NOT NULL,
    original_size INTEGER NOT NULL,
    compressed_size INTEGER NOT NULL,
    original_content TEXT NOT NULL,  -- JSON
    archived_events TEXT,
    FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
);

Configuration

INSERT INTO config (key, value) VALUES
    ('compact_tier1_days', '30'),
    ('compact_tier1_dep_levels', '2'),
    ('compact_tier2_days', '90'),
    ('compact_tier2_dep_levels', '5'),
    ('compact_tier2_commits', '100'),
    ('compact_model', 'claude-3-5-haiku-20241022'),
    ('compact_batch_size', '50'),
    ('compact_parallel_workers', '5'),
    ('auto_compact_enabled', 'false');

Haiku Prompts

Tier 1 (300 words)

Summarize this closed software issue. Preserve key decisions,
implementation approach, and outcome. Max 300 words.

Title: {{.Title}}
Type: {{.IssueType}}
Priority: {{.Priority}}

Description: {{.Description}}
Design Notes: {{.Design}}
Implementation Notes: {{.Notes}}
Acceptance Criteria: {{.AcceptanceCriteria}}

Output format:
**Summary:** [2-3 sentences: problem, solution, outcome]
**Key Decisions:** [bullet points of non-obvious choices]
**Resolution:** [how it was closed]

Tier 2 (150 words)

Ultra-compress this old closed issue to ≤150 words.
Focus on lasting architectural impact.

Title: {{.Title}}
Original Summary: {{.Description}}

Output a single paragraph covering:
- What was built/fixed
- Why it mattered
- Lasting impact (if any)

Success Metrics

• Space: 70-85% reduction (Tier 1), 90-95% (Tier 2)
• Quality: Summaries preserve essential context
• Safety: 100% restore success rate
• Performance: <100ms candidate query, ~2s per issue (Haiku)
• Cost: <$1.50 per 1,000 issues

Risks & Mitigations

Risk	Mitigation
Info loss in summaries	Dry-run review, restore capability, prompt tuning
API rate limits	Exponential backoff, respect limits, batch sizing
Accidental compression	Dry-run required, snapshots permanent
JSONL conflicts	Compaction is deterministic, git handles well

Next Steps

1. Review design: COMPACTION_DESIGN.md (comprehensive spec)
2. File issues: bd create -f compaction-issues.md (16 issues)
3. Start implementation: Begin with schema + config (Issues 1-2)
4. Iterate: Build MVP (Tier 1 only), then enhance

Questions?

• Full design: COMPACTION_DESIGN.md
• Issue breakdown: compaction-issues.md
• Reference implementation in Phase 1-4 above

---

Ready to implement in another session!