johno/beads
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add comprehensive compaction documentation

Browse Source 
- Updated README.md with Tier 1/2 info, restore command, cost analysis
- Created COMPACTION.md with full guide covering:
  - How compaction works (architecture, two-tier system)
  - CLI reference and examples
  - Eligibility rules and configuration
  - Cost analysis with detailed tables
  - Automation examples (cron, workflows)
  - Safety, recovery, and troubleshooting
  - FAQ and best practices
- Added examples/compaction/ with 3 scripts:
  - workflow.sh: Interactive compaction workflow
  - cron-compact.sh: Automated monthly compaction
  - auto-compact.sh: Smart threshold-based compaction
  - README.md: Examples documentation

Closes bd-265

Amp-Thread-ID: https://ampcode.com/threads/T-8113e88e-1cd0-4a9e-b581-07045a3ed31e
Co-authored-by: Amp <amp@ampcode.com>
This commit is contained in:
Steve Yegge
2025-10-16 01:09:48 -07:00
parent 6786d68365
commit 54469936a7
7 changed files with 913 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.beads/issues.jsonl
View File

@@ -165,7 +165,7 @@
{"id":"bd-248","title":"Test reopen command","description":"","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-15T16:28:44.246154-07:00","updated_at":"2025-10-16T01:00:55.664562-07:00","closed_at":"2025-10-15T17:05:23.644788-07:00"}
{"id":"bd-249","title":"Test reopen command","description":"","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-15T16:28:49.924381-07:00","updated_at":"2025-10-16T01:00:55.672993-07:00","closed_at":"2025-10-15T16:28:55.491141-07:00"}
{"id":"bd-25","title":"Add transaction support to storage layer for atomic multi-operation workflows","description":"Currently each storage method (CreateIssue, UpdateIssue, etc.) starts its own transaction. This makes it impossible to perform atomic multi-step operations like collision resolution. Add support for passing *sql.Tx through the storage interface, or create transaction-aware versions of methods. This would make remapCollisions and other batch operations truly atomic.","status":"closed","priority":4,"issue_type":"feature","created_at":"2025-10-14T14:43:06.910892-07:00","updated_at":"2025-10-16T01:00:55.67684-07:00","closed_at":"2025-10-15T03:01:29.570206-07:00"}
{"id":"bd-250","title":"Implement --format flag for bd list (from PR #46)","description":"PR #46 by tmc adds --format flag with Go template support for bd list, including presets for 'digraph' and 'dot' (Graphviz) output with status-based color coding. Unfortunately the PR is based on old main and would delete labels, reopen, and storage tests. Need to reimplement the feature atop current main.\n\nFeatures to implement:\n- --format flag for bd list\n- 'digraph' preset: basic 'from to' format for golang.org/x/tools/cmd/digraph\n- 'dot' preset: Graphviz compatible output with color-coded statuses\n- Custom Go template support with vars: IssueID, DependsOnID, Type, Issue, Dependency\n- Status-based colors: open=white, in_progress=lightyellow, blocked=lightcoral, closed=lightgray\n\nExamples:\n- bd list --format=digraph | digraph nodes\n- bd list --format=dot | dot -Tsvg -o deps.svg\n- bd list --format='{{.IssueID}} -\u003e {{.DependsOnID}} [{{.Type}}]'\n\nOriginal PR: https://github.com/steveyegge/beads/pull/46","status":"open","priority":2,"issue_type":"feature","created_at":"2025-10-15T21:13:11.6698-07:00","updated_at":"2025-10-16T01:00:55.679798-07:00","external_ref":"gh-46"}
{"id":"bd-250","title":"Implement --format flag for bd list (from PR #46)","description":"PR #46 by tmc adds --format flag with Go template support for bd list, including presets for 'digraph' and 'dot' (Graphviz) output with status-based color coding. Unfortunately the PR is based on old main and would delete labels, reopen, and storage tests. Need to reimplement the feature atop current main.\n\nFeatures to implement:\n- --format flag for bd list\n- 'digraph' preset: basic 'from to' format for golang.org/x/tools/cmd/digraph\n- 'dot' preset: Graphviz compatible output with color-coded statuses\n- Custom Go template support with vars: IssueID, DependsOnID, Type, Issue, Dependency\n- Status-based colors: open=white, in_progress=lightyellow, blocked=lightcoral, closed=lightgray\n\nExamples:\n- bd list --format=digraph | digraph nodes\n- bd list --format=dot | dot -Tsvg -o deps.svg\n- bd list --format='{{.IssueID}} -\u003e {{.DependsOnID}} [{{.Type}}]'\n\nOriginal PR: https://github.com/steveyegge/beads/pull/46","status":"closed","priority":2,"issue_type":"feature","created_at":"2025-10-15T21:13:11.6698-07:00","updated_at":"2025-10-16T01:06:09.691023-07:00","closed_at":"2025-10-16T01:06:09.691023-07:00","external_ref":"gh-46"}
{"id":"bd-251","title":"Epic: Add intelligent database compaction with Claude Haiku","description":"Implement multi-tier database compaction using Claude Haiku to semantically compress old, closed issues. This keeps the database lightweight and agent-friendly while preserving essential context.\n\nGoals:\n- 70-95% space reduction for eligible issues\n- Full restore capability via snapshots\n- Opt-in with dry-run safety\n- ~$1 per 1,000 issues compacted","acceptance_criteria":"- Schema migration with snapshots table\n- Haiku integration for summarization\n- Two-tier compaction (30d, 90d)\n- CLI with dry-run, restore, stats\n- Full test coverage\n- Documentation complete","status":"open","priority":2,"issue_type":"epic","created_at":"2025-10-15T21:51:23.210339-07:00","updated_at":"2025-10-16T01:00:55.68041-07:00"}
{"id":"bd-252","title":"Add compaction schema and migrations","description":"Add database schema support for issue compaction tracking and snapshot storage.","design":"Add three columns to `issues` table:\n- `compaction_level INTEGER DEFAULT 0` - 0=original, 1=tier1, 2=tier2\n- `compacted_at DATETIME` - when last compacted\n- `original_size INTEGER` - bytes before first compaction\n\nCreate `issue_snapshots` table:\n```sql\nCREATE TABLE issue_snapshots (\n id INTEGER PRIMARY KEY AUTOINCREMENT,\n issue_id TEXT NOT NULL,\n snapshot_time DATETIME NOT NULL,\n compaction_level INTEGER NOT NULL,\n original_size INTEGER NOT NULL,\n compressed_size INTEGER NOT NULL,\n original_content TEXT NOT NULL, -- JSON blob\n archived_events TEXT,\n FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE\n);\n```\n\nAdd indexes:\n- `idx_snapshots_issue` on `issue_id`\n- `idx_snapshots_level` on `compaction_level`\n\nAdd migration functions in `internal/storage/sqlite/sqlite.go`:\n- `migrateCompactionColumns(db *sql.DB) error`\n- `migrateSnapshotsTable(db *sql.DB) error`","acceptance_criteria":"- Existing databases migrate automatically\n- New databases include columns by default\n- Migration is idempotent (safe to run multiple times)\n- No data loss during migration\n- Tests verify migration on fresh and existing DBs","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-15T21:51:23.216371-07:00","updated_at":"2025-10-16T01:00:55.681294-07:00","closed_at":"2025-10-15T22:02:27.638283-07:00"}
{"id":"bd-253","title":"Add compaction configuration keys","description":"Add configuration keys for compaction behavior with sensible defaults.","design":"Add to `internal/storage/sqlite/schema.go` initial config:\n```sql\nINSERT OR IGNORE INTO config (key, value) VALUES\n ('compact_tier1_days', '30'),\n ('compact_tier1_dep_levels', '2'),\n ('compact_tier2_days', '90'),\n ('compact_tier2_dep_levels', '5'),\n ('compact_tier2_commits', '100'),\n ('compact_model', 'claude-3-5-haiku-20241022'),\n ('compact_batch_size', '50'),\n ('compact_parallel_workers', '5'),\n ('auto_compact_enabled', 'false');\n```\n\nAdd helper functions for loading config into typed struct.","acceptance_criteria":"- Config keys created on init\n- Existing DBs get defaults on migration\n- `bd config get/set` works with all keys\n- Type validation (days=int, enabled=bool)\n- Documentation in README.md","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-15T21:51:23.22391-07:00","updated_at":"2025-10-16T01:00:55.682647-07:00","closed_at":"2025-10-15T22:08:44.984927-07:00"}
@@ -181,7 +181,7 @@
{"id":"bd-262","title":"Add EventCompacted to event system","description":"Add new event type for tracking compaction in audit trail.","design":"1. Add to `internal/types/types.go`:\n```go\nconst EventCompacted EventType = \"compacted\"\n```\n\n2. Record event during compaction:\n```go\neventData := map[string]interface{}{\n \"tier\": tier,\n \"original_size\": originalSize,\n \"compressed_size\": compressedSize,\n \"reduction_pct\": (1 - float64(compressedSize)/float64(originalSize)) * 100,\n}\n```\n\n3. Update event display in `bd show`.","acceptance_criteria":"- Event includes tier, original_size, compressed_size, reduction_pct\n- Shows in event history (`bd events \u003cid\u003e`)\n- Exports to JSONL correctly\n- `bd show` displays compaction status and marker","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-15T21:51:23.244219-07:00","updated_at":"2025-10-16T01:00:55.693486-07:00","closed_at":"2025-10-16T00:59:17.465182-07:00"}
{"id":"bd-263","title":"Add compaction indicator to `bd show`","description":"Update `bd show` command to display compaction status prominently.","design":"Add to issue display:\n```\nbd-42: Fix authentication bug [CLOSED] 🗜️\n\nStatus: closed (compacted L1)\n...\n\n---\n💾 Restore: bd compact --restore bd-42\n📊 Original: 2,341 bytes | Compressed: 468 bytes (80% reduction)\n🗜️ Compacted: 2025-10-15 (Tier 1)\n```\n\nEmoji indicators:\n- Tier 1: 🗜️\n- Tier 2: 📦","acceptance_criteria":"- Compaction status visible in title line\n- Footer shows size savings when compacted\n- Restore command shown for compacted issues\n- Works with `--json` output (includes compaction fields)\n- Emoji optional (controlled by config or terminal detection)","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-15T21:51:23.253091-07:00","updated_at":"2025-10-16T01:03:59.28912-07:00","closed_at":"2025-10-16T01:03:59.28912-07:00"}
{"id":"bd-264","title":"Write compaction tests","description":"Comprehensive test suite for compaction functionality.","design":"Test coverage:\n\n1. **Candidate Identification:**\n - Eligibility by time\n - Dependency depth checking\n - Mixed status dependents\n - Edge cases (no deps, circular deps)\n\n2. **Snapshots:**\n - Create and restore\n - Multiple snapshots per issue\n - Content integrity (UTF-8, special chars)\n\n3. **Tier 1 Compaction:**\n - Single issue compaction\n - Batch processing\n - Error handling (API failures)\n\n4. **Tier 2 Compaction:**\n - Requires Tier 1\n - Events pruning\n - Commit counting fallback\n\n5. **CLI:**\n - All flag combinations\n - Dry-run accuracy\n - JSON output parsing\n\n6. **Integration:**\n - End-to-end flow\n - JSONL export/import\n - Restore verification","acceptance_criteria":"- Test coverage \u003e80%\n- All edge cases covered\n- Mock Haiku API in tests (no real API calls)\n- Integration tests pass\n- `go test ./...` passes\n- Benchmarks for performance-critical paths","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-15T21:51:23.262504-07:00","updated_at":"2025-10-16T01:00:55.694731-07:00","closed_at":"2025-10-16T00:02:11.246331-07:00"}
{"id":"bd-265","title":"Add compaction documentation","description":"Document compaction feature in README and create detailed COMPACTION.md guide.","design":"**Update README.md:**\n- Add to Features section\n- CLI examples (dry-run, compact, restore, stats)\n- Configuration guide\n- Cost analysis\n\n**Create COMPACTION.md:**\n- How compaction works (architecture overview)\n- When to use each tier\n- Detailed cost analysis with examples\n- Safety mechanisms (snapshots, restore, dry-run)\n- Troubleshooting guide\n- FAQ\n\n**Create examples/compaction/:**\n- `workflow.sh` - Example monthly compaction workflow\n- `cron-compact.sh` - Cron job setup\n- `auto-compact.sh` - Auto-compaction script","acceptance_criteria":"- README.md updated with compaction section\n- COMPACTION.md comprehensive and clear\n- Examples work as documented (tested)\n- Screenshots or ASCII examples included\n- API key setup documented (env var vs config)\n- Covers common questions and issues","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-15T21:51:23.265589-07:00","updated_at":"2025-10-16T01:00:55.695187-07:00"}
{"id":"bd-265","title":"Add compaction documentation","description":"Document compaction feature in README and create detailed COMPACTION.md guide.","design":"**Update README.md:**\n- Add to Features section\n- CLI examples (dry-run, compact, restore, stats)\n- Configuration guide\n- Cost analysis\n\n**Create COMPACTION.md:**\n- How compaction works (architecture overview)\n- When to use each tier\n- Detailed cost analysis with examples\n- Safety mechanisms (snapshots, restore, dry-run)\n- Troubleshooting guide\n- FAQ\n\n**Create examples/compaction/:**\n- `workflow.sh` - Example monthly compaction workflow\n- `cron-compact.sh` - Cron job setup\n- `auto-compact.sh` - Auto-compaction script","acceptance_criteria":"- README.md updated with compaction section\n- COMPACTION.md comprehensive and clear\n- Examples work as documented (tested)\n- Screenshots or ASCII examples included\n- API key setup documented (env var vs config)\n- Covers common questions and issues","status":"in_progress","priority":2,"issue_type":"task","created_at":"2025-10-15T21:51:23.265589-07:00","updated_at":"2025-10-16T01:06:45.804764-07:00"}
{"id":"bd-266","title":"Optional: Implement auto-compaction","description":"Implement automatic compaction triggered by certain operations when enabled via config.","design":"Trigger points (when `auto_compact_enabled = true`):\n1. `bd stats` - check and compact if candidates exist\n2. `bd export` - before exporting\n3. Configurable: on any read operation after N candidates accumulate\n\nAdd:\n```go\nfunc (s *SQLiteStorage) AutoCompact(ctx context.Context) error {\n enabled, _ := s.GetConfig(ctx, \"auto_compact_enabled\")\n if enabled != \"true\" {\n return nil\n }\n\n // Run Tier 1 compaction on all candidates\n // Limit to batch_size to avoid long operations\n // Log activity for transparency\n}\n```","acceptance_criteria":"- Respects auto_compact_enabled config (default: false)\n- Limits batch size to avoid blocking operations\n- Logs compaction activity (visible with --verbose)\n- Can be disabled per-command with `--no-auto-compact` flag\n- Only compacts Tier 1 (Tier 2 remains manual)\n- Doesn't run more than once per hour (rate limiting)","status":"open","priority":3,"issue_type":"task","created_at":"2025-10-15T21:51:23.281006-07:00","updated_at":"2025-10-16T01:00:55.695589-07:00"}
{"id":"bd-267","title":"Optional: Add git commit counting","description":"Implement git commit counting for \"project time\" measurement as alternative to calendar time for Tier 2 eligibility.","design":"```go\nfunc getCommitsSince(closedAt time.Time) (int, error) {\n cmd := exec.Command(\"git\", \"rev-list\", \"--count\",\n fmt.Sprintf(\"--since=%s\", closedAt.Format(time.RFC3339)), \"HEAD\")\n output, err := cmd.Output()\n if err != nil {\n return 0, err // Not in git repo or git not available\n }\n return strconv.Atoi(strings.TrimSpace(string(output)))\n}\n```\n\nFallback strategies:\n1. Git commit count (preferred)\n2. Issue counter delta (store counter at close time, compare later)\n3. Pure time-based (90 days)","acceptance_criteria":"- Counts commits since closed_at timestamp\n- Handles git not available gracefully (falls back)\n- Fallback to issue counter delta works\n- Configurable via compact_tier2_commits config key\n- Tested with real git repo\n- Works in non-git environments","status":"open","priority":3,"issue_type":"task","created_at":"2025-10-15T21:51:23.284781-07:00","updated_at":"2025-10-16T01:00:55.695972-07:00"}
{"id":"bd-268","title":"Explore in-memory embedded SQL alternatives to SQLite","description":"Investigate lightweight in-memory embedded SQL databases as alternative backends for environments where SQLite is problematic or considered too heavyweight. This would provide flexibility for different deployment scenarios.","design":"Research options:\n- modernc.org/sqlite (pure Go SQLite implementation, no cgo)\n- rqlite (distributed SQLite with Raft)\n- go-memdb (in-memory database by HashiCorp)\n- badger (embedded key-value store, would need SQL layer)\n- bbolt (embedded key-value store)\n- duckdb (lightweight analytical database)\n\nEvaluate on:\n- Memory footprint vs SQLite\n- cgo dependency (pure Go preferred)\n- SQL compatibility level\n- Transaction support\n- Performance characteristics\n- Maintenance/community status\n- Migration complexity from SQLite\n\nConsider creating a storage abstraction layer to support multiple backends.","acceptance_criteria":"- Document comparison of at least 3 alternatives\n- Benchmark memory usage and performance vs SQLite\n- Assess migration effort for each option\n- Recommendation on whether to support alternatives\n- If yes, prototype storage interface abstraction","notes":"Worth noting: modernc.org/sqlite is a pure Go implementation (no cgo) that might already address the \"heavyweight\" concern, since much of SQLite's overhead comes from cgo calls. Should evaluate this first before exploring completely different database technologies.","status":"open","priority":3,"issue_type":"task","created_at":"2025-10-15T23:17:33.560045-07:00","updated_at":"2025-10-16T01:00:55.696359-07:00"}