Add compaction feature design and file 17 issues (bd-251 to bd-267)
Amp-Thread-ID: https://ampcode.com/threads/T-8535178e-f814-43e7-a8a0-4aea93ef3970 Co-authored-by: Amp <amp@ampcode.com>
This commit is contained in:
Steve Yegge
@@ -166,7 +166,24 @@
|
||||
{"id":"bd-249","title":"Test reopen command","description":"","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-15T16:28:49.924381-07:00","updated_at":"2025-10-15T16:28:55.491141-07:00","closed_at":"2025-10-15T16:28:55.491141-07:00"}
|
||||
{"id":"bd-25","title":"Add transaction support to storage layer for atomic multi-operation workflows","description":"Currently each storage method (CreateIssue, UpdateIssue, etc.) starts its own transaction. This makes it impossible to perform atomic multi-step operations like collision resolution. Add support for passing *sql.Tx through the storage interface, or create transaction-aware versions of methods. This would make remapCollisions and other batch operations truly atomic.","status":"closed","priority":4,"issue_type":"feature","created_at":"2025-10-14T14:43:06.910892-07:00","updated_at":"2025-10-15T16:27:22.001363-07:00","closed_at":"2025-10-15T03:01:29.570206-07:00"}
|
||||
{"id":"bd-250","title":"Implement --format flag for bd list (from PR #46)","description":"PR #46 by tmc adds --format flag with Go template support for bd list, including presets for 'digraph' and 'dot' (Graphviz) output with status-based color coding. Unfortunately the PR is based on old main and would delete labels, reopen, and storage tests. Need to reimplement the feature atop current main.\n\nFeatures to implement:\n- --format flag for bd list\n- 'digraph' preset: basic 'from to' format for golang.org/x/tools/cmd/digraph\n- 'dot' preset: Graphviz compatible output with color-coded statuses\n- Custom Go template support with vars: IssueID, DependsOnID, Type, Issue, Dependency\n- Status-based colors: open=white, in_progress=lightyellow, blocked=lightcoral, closed=lightgray\n\nExamples:\n- bd list --format=digraph | digraph nodes\n- bd list --format=dot | dot -Tsvg -o deps.svg\n- bd list --format='{{.IssueID}} -\u003e {{.DependsOnID}} [{{.Type}}]'\n\nOriginal PR: https://github.com/steveyegge/beads/pull/46","status":"open","priority":2,"issue_type":"feature","created_at":"2025-10-15T21:13:11.6698-07:00","updated_at":"2025-10-15T21:13:11.6698-07:00","external_ref":"gh-46"}
|
||||
{"id":"bd-251","title":"Epic: Add intelligent database compaction with Claude Haiku","description":"Implement multi-tier database compaction using Claude Haiku to semantically compress old, closed issues. This keeps the database lightweight and agent-friendly while preserving essential context.\n\nGoals:\n- 70-95% space reduction for eligible issues\n- Full restore capability via snapshots\n- Opt-in with dry-run safety\n- ~$1 per 1,000 issues compacted","acceptance_criteria":"- Schema migration with snapshots table\n- Haiku integration for summarization\n- Two-tier compaction (30d, 90d)\n- CLI with dry-run, restore, stats\n- Full test coverage\n- Documentation complete","status":"open","priority":2,"issue_type":"epic","created_at":"2025-10-15T21:51:23.210339-07:00","updated_at":"2025-10-15T21:51:23.210339-07:00","labels":["---","compaction","epic","haiku","v1.1"]}
|
||||
{"id":"bd-252","title":"Add compaction schema and migrations","description":"Add database schema support for issue compaction tracking and snapshot storage.","design":"Add three columns to `issues` table:\n- `compaction_level INTEGER DEFAULT 0` - 0=original, 1=tier1, 2=tier2\n- `compacted_at DATETIME` - when last compacted\n- `original_size INTEGER` - bytes before first compaction\n\nCreate `issue_snapshots` table:\n```sql\nCREATE TABLE issue_snapshots (\n id INTEGER PRIMARY KEY AUTOINCREMENT,\n issue_id TEXT NOT NULL,\n snapshot_time DATETIME NOT NULL,\n compaction_level INTEGER NOT NULL,\n original_size INTEGER NOT NULL,\n compressed_size INTEGER NOT NULL,\n original_content TEXT NOT NULL, -- JSON blob\n archived_events TEXT,\n FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE\n);\n```\n\nAdd indexes:\n- `idx_snapshots_issue` on `issue_id`\n- `idx_snapshots_level` on `compaction_level`\n\nAdd migration functions in `internal/storage/sqlite/sqlite.go`:\n- `migrateCompactionColumns(db *sql.DB) error`\n- `migrateSnapshotsTable(db *sql.DB) error`","acceptance_criteria":"- Existing databases migrate automatically\n- New databases include columns by default\n- Migration is idempotent (safe to run multiple times)\n- No data loss during migration\n- Tests verify migration on fresh and existing DBs","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-15T21:51:23.216371-07:00","updated_at":"2025-10-15T21:51:23.216371-07:00","labels":["---","compaction","database","migration","schema"]}
|
||||
{"id":"bd-253","title":"Add compaction configuration keys","description":"Add configuration keys for compaction behavior with sensible defaults.","design":"Add to `internal/storage/sqlite/schema.go` initial config:\n```sql\nINSERT OR IGNORE INTO config (key, value) VALUES\n ('compact_tier1_days', '30'),\n ('compact_tier1_dep_levels', '2'),\n ('compact_tier2_days', '90'),\n ('compact_tier2_dep_levels', '5'),\n ('compact_tier2_commits', '100'),\n ('compact_model', 'claude-3-5-haiku-20241022'),\n ('compact_batch_size', '50'),\n ('compact_parallel_workers', '5'),\n ('auto_compact_enabled', 'false');\n```\n\nAdd helper functions for loading config into typed struct.","acceptance_criteria":"- Config keys created on init\n- Existing DBs get defaults on migration\n- `bd config get/set` works with all keys\n- Type validation (days=int, enabled=bool)\n- Documentation in README.md","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-15T21:51:23.22391-07:00","updated_at":"2025-10-15T21:51:23.22391-07:00","labels":["---","compaction","config","configuration"]}
|
||||
{"id":"bd-254","title":"Implement candidate identification queries","description":"Write SQL queries to identify issues eligible for Tier 1 and Tier 2 compaction based on closure time and dependency status.","design":"Create `internal/storage/sqlite/compact.go` with:\n\n```go\ntype CompactionCandidate struct {\n IssueID string\n ClosedAt time.Time\n OriginalSize int\n EstimatedSize int\n DependentCount int\n}\n\nfunc (s *SQLiteStorage) GetTier1Candidates(ctx context.Context) ([]*CompactionCandidate, error)\nfunc (s *SQLiteStorage) GetTier2Candidates(ctx context.Context) ([]*CompactionCandidate, error)\nfunc (s *SQLiteStorage) CheckEligibility(ctx context.Context, issueID string, tier int) (bool, string, error)\n```\n\nUse recursive CTE for dependency depth checking (similar to ready_issues view).","acceptance_criteria":"- Tier 1 query filters by days and dependency depth\n- Tier 2 query includes commit/issue count checks\n- Dependency checking handles circular deps gracefully\n- Performance: \u003c100ms for 10,000 issue database\n- Tests cover edge cases (no deps, circular deps, mixed status)","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-15T21:51:23.225835-07:00","updated_at":"2025-10-15T21:51:23.225835-07:00","labels":["---","compaction","dependencies","query","sql"]}
|
||||
{"id":"bd-255","title":"Create Haiku client and prompt templates","description":"Implement Claude Haiku API client with template-based prompts for Tier 1 and Tier 2 summarization.","design":"Create `internal/compact/haiku.go`:\n\n```go\ntype HaikuClient struct {\n client *anthropic.Client\n model string\n}\n\nfunc NewHaikuClient(apiKey string) (*HaikuClient, error)\nfunc (h *HaikuClient) SummarizeTier1(ctx context.Context, issue *types.Issue) (string, error)\nfunc (h *HaikuClient) SummarizeTier2(ctx context.Context, issue *types.Issue) (string, error)\n```\n\nUse text/template for prompt rendering.\n\nTier 1 output format:\n```\n**Summary:** [2-3 sentences]\n**Key Decisions:** [bullet points]\n**Resolution:** [outcome]\n```\n\nTier 2 output format:\n```\nSingle paragraph ≤150 words covering what was built, why it mattered, lasting impact.\n```","acceptance_criteria":"- API key from env var or config (env takes precedence)\n- Prompts render correctly with templates\n- Rate limiting handled gracefully (exponential backoff)\n- Network errors retry up to 3 times\n- Mock tests for API calls","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-15T21:51:23.229702-07:00","updated_at":"2025-10-15T21:51:23.229702-07:00","labels":["---","api","compaction","haiku","llm"]}
|
||||
{"id":"bd-256","title":"Implement snapshot creation and restoration","description":"Implement snapshot creation before compaction and restoration capability to undo compaction.","design":"Add to `internal/storage/sqlite/compact.go`:\n\n```go\nfunc (s *SQLiteStorage) CreateSnapshot(ctx context.Context, issue *types.Issue, level int) error\nfunc (s *SQLiteStorage) RestoreFromSnapshot(ctx context.Context, issueID string, level int) error\nfunc (s *SQLiteStorage) GetSnapshots(ctx context.Context, issueID string) ([]*Snapshot, error)\n```\n\nSnapshot JSON structure:\n```json\n{\n \"description\": \"...\",\n \"design\": \"...\",\n \"notes\": \"...\",\n \"acceptance_criteria\": \"...\",\n \"title\": \"...\"\n}\n```","acceptance_criteria":"- Snapshot created atomically with compaction\n- Restore returns exact original content\n- Multiple snapshots per issue supported (Tier 1 → Tier 2)\n- JSON encoding handles UTF-8 and special characters\n- Size calculation is accurate (UTF-8 bytes)","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-15T21:51:23.231906-07:00","updated_at":"2025-10-15T21:51:23.231906-07:00","labels":["---","compaction","restore","safety","snapshot"]}
|
||||
{"id":"bd-257","title":"Implement Tier 1 compaction logic","description":"Implement the core Tier 1 compaction process: snapshot → summarize → update.","design":"Add to `internal/compact/compactor.go`:\n\n```go\ntype Compactor struct {\n store storage.Storage\n haiku *HaikuClient\n config *CompactConfig\n}\n\nfunc New(store storage.Storage, apiKey string, config *CompactConfig) (*Compactor, error)\nfunc (c *Compactor) CompactTier1(ctx context.Context, issueID string) error\nfunc (c *Compactor) CompactTier1Batch(ctx context.Context, issueIDs []string) error\n```\n\nProcess:\n1. Verify eligibility\n2. Calculate original size\n3. Create snapshot\n4. Call Haiku for summary\n5. Update issue (description=summary, clear design/notes/criteria)\n6. Set compaction_level=1, compacted_at=now, original_size\n7. Record EventCompacted\n8. Mark dirty for export","acceptance_criteria":"- Single issue compaction works end-to-end\n- Batch processing with parallel workers (5 concurrent)\n- Errors don't corrupt database (transaction rollback)\n- EventCompacted includes size savings\n- Dry-run mode (identify + size estimate only, no API calls)","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-15T21:51:23.23391-07:00","updated_at":"2025-10-15T21:51:23.23391-07:00","labels":["---","compaction","core-logic","tier1"]}
|
||||
{"id":"bd-258","title":"Implement Tier 2 compaction logic","description":"Implement Tier 2 ultra-compression: more aggressive summarization and optional event pruning.","design":"Add to `internal/compact/compactor.go`:\n\n```go\nfunc (c *Compactor) CompactTier2(ctx context.Context, issueID string) error\nfunc (c *Compactor) CompactTier2Batch(ctx context.Context, issueIDs []string) error\n```\n\nProcess:\n1. Verify issue is at compaction_level = 1\n2. Check Tier 2 eligibility (days, deps, commits/issues)\n3. Create Tier 2 snapshot\n4. Call Haiku with ultra-compression prompt\n5. Update issue (description = single paragraph, clear all other fields)\n6. Set compaction_level = 2\n7. Optionally prune events (keep created/closed, archive rest to snapshot)","acceptance_criteria":"- Requires existing Tier 1 compaction\n- Git commit counting works (with fallback to issue counter)\n- Events optionally pruned (config: compact_events_enabled)\n- Archived events stored in snapshot JSON\n- Size reduction 90-95%","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-15T21:51:23.23586-07:00","updated_at":"2025-10-15T21:51:23.23586-07:00","labels":["---","advanced","compaction","tier2"]}
|
||||
{"id":"bd-259","title":"Add `bd compact` CLI command","description":"Implement the `bd compact` command with dry-run, batch processing, and progress reporting.","design":"Create `cmd/bd/compact.go`:\n\n```go\nvar compactCmd = \u0026cobra.Command{\n Use: \"compact\",\n Short: \"Compact old closed issues to save space\",\n}\n\nFlags:\n --dry-run Preview without compacting\n --tier int Compaction tier (1 or 2, default: 1)\n --all Process all candidates\n --id string Compact specific issue\n --force Force compact (bypass checks, requires --id)\n --batch-size int Issues per batch\n --workers int Parallel workers\n --json JSON output\n```","acceptance_criteria":"- `--dry-run` shows accurate preview with size estimates\n- `--all` processes all candidates\n- `--id` compacts single issue\n- `--force` bypasses eligibility checks (only with --id)\n- Progress bar for batches (e.g., [████████] 47/47)\n- JSON output with `--json`\n- Exit codes: 0=success, 1=error\n- Shows summary: count, size saved, cost, time","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-15T21:51:23.238373-07:00","updated_at":"2025-10-15T21:51:23.238373-07:00","labels":["---","cli","command","compaction"]}
|
||||
{"id":"bd-26","title":"Optimize reference updates to avoid loading all issues into memory","description":"In updateReferences(), we call SearchIssues with no filter to get ALL issues for updating references. For large databases (10k+ issues), this loads everything into memory. Options: 1) Use batched processing with LIMIT/OFFSET, 2) Use SQL UPDATE with REPLACE() directly, 3) Stream results instead of loading all at once. Located in collision.go:266","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-14T14:43:06.911497-07:00","updated_at":"2025-10-15T16:27:22.001829-07:00"}
|
||||
{"id":"bd-260","title":"Add `bd compact --restore` functionality","description":"Implement restore command to undo compaction from snapshots.","design":"Add to `cmd/bd/compact.go`:\n\n```go\nvar compactRestore string\n\ncompactCmd.Flags().StringVar(\u0026compactRestore, \"restore\", \"\", \"Restore issue from snapshot\")\n```\n\nProcess:\n1. Load snapshot for issue\n2. Parse JSON content\n3. Update issue with original content\n4. Set compaction_level = 0, compacted_at = NULL, original_size = NULL\n5. Record event (EventRestored or EventUpdated)\n6. Mark dirty for export","acceptance_criteria":"- Restores exact original content\n- Handles multiple snapshots (use latest by default)\n- `--level` flag to choose specific snapshot\n- Updates compaction_level correctly\n- Exports restored content to JSONL\n- Shows before/after in output","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-15T21:51:23.240267-07:00","updated_at":"2025-10-15T21:51:23.240267-07:00","labels":["---","cli","compaction","restore"]}
|
||||
{"id":"bd-261","title":"Add `bd compact --stats` command","description":"Add statistics command showing compaction status and potential savings.","design":"```go\nvar compactStats bool\n\ncompactCmd.Flags().BoolVar(\u0026compactStats, \"stats\", false, \"Show compaction statistics\")\n```\n\nOutput:\n- Total issues, by compaction level (0, 1, 2)\n- Current DB size vs estimated uncompacted size\n- Space savings (KB/MB and %)\n- Candidates for each tier with size estimates\n- Estimated API cost (Haiku pricing)","acceptance_criteria":"- Accurate counts by compaction_level\n- Size calculations include all text fields (UTF-8 bytes)\n- Shows candidates with eligibility reasons\n- Cost estimation based on current Haiku pricing\n- JSON output supported\n- Clear, readable table format","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-15T21:51:23.242041-07:00","updated_at":"2025-10-15T21:51:23.242041-07:00","labels":["---","compaction","reporting","stats"]}
|
||||
{"id":"bd-262","title":"Add EventCompacted to event system","description":"Add new event type for tracking compaction in audit trail.","design":"1. Add to `internal/types/types.go`:\n```go\nconst EventCompacted EventType = \"compacted\"\n```\n\n2. Record event during compaction:\n```go\neventData := map[string]interface{}{\n \"tier\": tier,\n \"original_size\": originalSize,\n \"compressed_size\": compressedSize,\n \"reduction_pct\": (1 - float64(compressedSize)/float64(originalSize)) * 100,\n}\n```\n\n3. Update event display in `bd show`.","acceptance_criteria":"- Event includes tier, original_size, compressed_size, reduction_pct\n- Shows in event history (`bd events \u003cid\u003e`)\n- Exports to JSONL correctly\n- `bd show` displays compaction status and marker","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-15T21:51:23.244219-07:00","updated_at":"2025-10-15T21:51:23.244219-07:00","labels":["---","audit","compaction","events"]}
|
||||
{"id":"bd-263","title":"Add compaction indicator to `bd show`","description":"Update `bd show` command to display compaction status prominently.","design":"Add to issue display:\n```\nbd-42: Fix authentication bug [CLOSED] 🗜️\n\nStatus: closed (compacted L1)\n...\n\n---\n💾 Restore: bd compact --restore bd-42\n📊 Original: 2,341 bytes | Compressed: 468 bytes (80% reduction)\n🗜️ Compacted: 2025-10-15 (Tier 1)\n```\n\nEmoji indicators:\n- Tier 1: 🗜️\n- Tier 2: 📦","acceptance_criteria":"- Compaction status visible in title line\n- Footer shows size savings when compacted\n- Restore command shown for compacted issues\n- Works with `--json` output (includes compaction fields)\n- Emoji optional (controlled by config or terminal detection)","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-15T21:51:23.253091-07:00","updated_at":"2025-10-15T21:51:23.253091-07:00","labels":["---","compaction","display","ui"]}
|
||||
{"id":"bd-264","title":"Write compaction tests","description":"Comprehensive test suite for compaction functionality.","design":"Test coverage:\n\n1. **Candidate Identification:**\n - Eligibility by time\n - Dependency depth checking\n - Mixed status dependents\n - Edge cases (no deps, circular deps)\n\n2. **Snapshots:**\n - Create and restore\n - Multiple snapshots per issue\n - Content integrity (UTF-8, special chars)\n\n3. **Tier 1 Compaction:**\n - Single issue compaction\n - Batch processing\n - Error handling (API failures)\n\n4. **Tier 2 Compaction:**\n - Requires Tier 1\n - Events pruning\n - Commit counting fallback\n\n5. **CLI:**\n - All flag combinations\n - Dry-run accuracy\n - JSON output parsing\n\n6. **Integration:**\n - End-to-end flow\n - JSONL export/import\n - Restore verification","acceptance_criteria":"- Test coverage \u003e80%\n- All edge cases covered\n- Mock Haiku API in tests (no real API calls)\n- Integration tests pass\n- `go test ./...` passes\n- Benchmarks for performance-critical paths","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-15T21:51:23.262504-07:00","updated_at":"2025-10-15T21:51:23.262504-07:00","labels":["---","compaction","quality","testing"]}
|
||||
{"id":"bd-265","title":"Add compaction documentation","description":"Document compaction feature in README and create detailed COMPACTION.md guide.","design":"**Update README.md:**\n- Add to Features section\n- CLI examples (dry-run, compact, restore, stats)\n- Configuration guide\n- Cost analysis\n\n**Create COMPACTION.md:**\n- How compaction works (architecture overview)\n- When to use each tier\n- Detailed cost analysis with examples\n- Safety mechanisms (snapshots, restore, dry-run)\n- Troubleshooting guide\n- FAQ\n\n**Create examples/compaction/:**\n- `workflow.sh` - Example monthly compaction workflow\n- `cron-compact.sh` - Cron job setup\n- `auto-compact.sh` - Auto-compaction script","acceptance_criteria":"- README.md updated with compaction section\n- COMPACTION.md comprehensive and clear\n- Examples work as documented (tested)\n- Screenshots or ASCII examples included\n- API key setup documented (env var vs config)\n- Covers common questions and issues","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-15T21:51:23.265589-07:00","updated_at":"2025-10-15T21:51:23.265589-07:00","labels":["---","compaction","docs","documentation","examples"]}
|
||||
{"id":"bd-266","title":"Optional: Implement auto-compaction","description":"Implement automatic compaction triggered by certain operations when enabled via config.","design":"Trigger points (when `auto_compact_enabled = true`):\n1. `bd stats` - check and compact if candidates exist\n2. `bd export` - before exporting\n3. Configurable: on any read operation after N candidates accumulate\n\nAdd:\n```go\nfunc (s *SQLiteStorage) AutoCompact(ctx context.Context) error {\n enabled, _ := s.GetConfig(ctx, \"auto_compact_enabled\")\n if enabled != \"true\" {\n return nil\n }\n\n // Run Tier 1 compaction on all candidates\n // Limit to batch_size to avoid long operations\n // Log activity for transparency\n}\n```","acceptance_criteria":"- Respects auto_compact_enabled config (default: false)\n- Limits batch size to avoid blocking operations\n- Logs compaction activity (visible with --verbose)\n- Can be disabled per-command with `--no-auto-compact` flag\n- Only compacts Tier 1 (Tier 2 remains manual)\n- Doesn't run more than once per hour (rate limiting)","status":"open","priority":3,"issue_type":"task","created_at":"2025-10-15T21:51:23.281006-07:00","updated_at":"2025-10-15T21:51:23.281006-07:00","labels":["---","automation","compaction","optional","v1.2"]}
|
||||
{"id":"bd-267","title":"Optional: Add git commit counting","description":"Implement git commit counting for \"project time\" measurement as alternative to calendar time for Tier 2 eligibility.","design":"```go\nfunc getCommitsSince(closedAt time.Time) (int, error) {\n cmd := exec.Command(\"git\", \"rev-list\", \"--count\",\n fmt.Sprintf(\"--since=%s\", closedAt.Format(time.RFC3339)), \"HEAD\")\n output, err := cmd.Output()\n if err != nil {\n return 0, err // Not in git repo or git not available\n }\n return strconv.Atoi(strings.TrimSpace(string(output)))\n}\n```\n\nFallback strategies:\n1. Git commit count (preferred)\n2. Issue counter delta (store counter at close time, compare later)\n3. Pure time-based (90 days)","acceptance_criteria":"- Counts commits since closed_at timestamp\n- Handles git not available gracefully (falls back)\n- Fallback to issue counter delta works\n- Configurable via compact_tier2_commits config key\n- Tested with real git repo\n- Works in non-git environments","status":"open","priority":3,"issue_type":"task","created_at":"2025-10-15T21:51:23.284781-07:00","updated_at":"2025-10-15T21:51:23.284781-07:00","labels":["compaction","git","optional","tier2"]}
|
||||
{"id":"bd-27","title":"Cache compiled regexes in replaceIDReferences for performance","description":"replaceIDReferences() compiles the same regex patterns on every call. With 100 issues and 10 ID mappings, that's 1000 regex compilations. Pre-compile regexes once and reuse. Can use a struct with compiled regex, placeholder, and newID. Located in collision.go:329. Estimated performance improvement: 10-100x for large batches.","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-14T14:43:06.911892-07:00","updated_at":"2025-10-15T16:27:22.002496-07:00","closed_at":"2025-10-15T03:01:29.570955-07:00"}
|
||||
{"id":"bd-28","title":"Improve error handling in dependency removal during remapping","description":"In updateDependencyReferences(), RemoveDependency errors are caught and ignored with continue (line 392). Comment says 'if dependency doesn't exist' but this catches ALL errors including real failures. Should check error type with errors.Is(err, ErrDependencyNotFound) and only ignore not-found errors, returning other errors properly.","status":"open","priority":3,"issue_type":"bug","created_at":"2025-10-14T14:43:06.912228-07:00","updated_at":"2025-10-15T16:27:22.003145-07:00"}
|
||||
{"id":"bd-29","title":"Use safer placeholder pattern in replaceIDReferences","description":"Currently uses __PLACEHOLDER_0__ which could theoretically collide with user text. Use a truly unique placeholder like null bytes: \\x00REMAP\\x00_0_\\x00 which are unlikely to appear in normal text. Located in collision.go:324. Very low probability issue but worth fixing for completeness.","status":"open","priority":3,"issue_type":"task","created_at":"2025-10-14T14:43:06.912567-07:00","updated_at":"2025-10-15T16:27:22.003668-07:00"}
|
||||
|
||||
1654
COMPACTION_DESIGN.md
Normal file
1654
COMPACTION_DESIGN.md
Normal file
File diff suppressed because it is too large
Load Diff
285
COMPACTION_SUMMARY.md
Normal file
285
COMPACTION_SUMMARY.md
Normal file
@@ -0,0 +1,285 @@
|
||||
# Issue Compaction - Quick Reference
|
||||
|
||||
**Status:** Design Complete, Ready for Implementation
|
||||
**Target:** Beads v1.1
|
||||
**Estimated Effort:** 40-60 hours (MVP)
|
||||
|
||||
## What Is This?
|
||||
|
||||
Add intelligent database compaction to beads using Claude Haiku to semantically compress old, closed issues. Keep your database lightweight while preserving the meaning of past work.
|
||||
|
||||
## Why?
|
||||
|
||||
- **Agent Efficiency:** Smaller DBs → faster queries → clearer thinking
|
||||
- **Context Management:** Agents need summaries, not verbose details
|
||||
- **Pragmatic:** Most work is throwaway; forensic value decays exponentially
|
||||
- **Git-Friendly:** Smaller JSONL exports
|
||||
|
||||
## How It Works
|
||||
|
||||
### Two-Tier Compression
|
||||
|
||||
**Tier 1: Standard (30 days)**
|
||||
- Closed ≥30 days, all dependents closed (2 levels deep)
|
||||
- Haiku summarizes to ~300 words
|
||||
- Keeps: title, summary with key decisions
|
||||
- Clears: verbose design/notes/criteria
|
||||
- **Result:** 70-85% space reduction
|
||||
|
||||
**Tier 2: Aggressive (90 days)**
|
||||
- Already Tier 1 compressed
|
||||
- Closed ≥90 days, deep dependencies closed (5 levels)
|
||||
- ≥100 git commits since closure (measures "project time")
|
||||
- Ultra-compress to single ≤150 word paragraph
|
||||
- Optionally prunes old events
|
||||
- **Result:** 90-95% space reduction
|
||||
|
||||
### Safety First
|
||||
|
||||
- **Snapshots:** Full original content saved before compaction
|
||||
- **Restore:** `bd compact --restore <id>` undoes compaction
|
||||
- **Dry-Run:** `bd compact --dry-run` previews without changing anything
|
||||
- **Git Backup:** JSONL commits preserve full history
|
||||
- **Opt-In:** Disabled by default (`auto_compact_enabled = false`)
|
||||
|
||||
## CLI
|
||||
|
||||
```bash
|
||||
# Preview candidates
|
||||
bd compact --dry-run
|
||||
|
||||
# Compact Tier 1
|
||||
bd compact --all
|
||||
|
||||
# Compact Tier 2
|
||||
bd compact --tier 2 --all
|
||||
|
||||
# Compact specific issue
|
||||
bd compact --id bd-42
|
||||
|
||||
# Restore compacted issue
|
||||
bd compact --restore bd-42
|
||||
|
||||
# Show statistics
|
||||
bd compact --stats
|
||||
```
|
||||
|
||||
## Cost
|
||||
|
||||
**Haiku Pricing:**
|
||||
- ~$0.0008 per issue (Tier 1)
|
||||
- ~$0.0003 per issue (Tier 2)
|
||||
- **~$1.10 per 1,000 issues total**
|
||||
|
||||
Negligible for typical usage (~$0.05/month for active project).
|
||||
|
||||
## Example Output
|
||||
|
||||
### Before Compaction
|
||||
```
|
||||
bd-42: Fix authentication bug [CLOSED]
|
||||
|
||||
Description: (800 words about the problem)
|
||||
|
||||
Design: (1,200 words of implementation notes)
|
||||
|
||||
Notes: (500 words of testing/deployment details)
|
||||
|
||||
Acceptance Criteria: (300 words of requirements)
|
||||
|
||||
Total: 2,341 bytes
|
||||
```
|
||||
|
||||
### After Tier 1 Compaction
|
||||
```
|
||||
bd-42: Fix authentication bug [CLOSED] 🗜️
|
||||
|
||||
**Summary:** Fixed race condition in JWT token refresh logic causing
|
||||
intermittent 401 errors. Implemented mutex-based locking. All users
|
||||
can now stay authenticated reliably.
|
||||
|
||||
**Key Decisions:**
|
||||
- Used sync.RWMutex for simpler reasoning
|
||||
- Added exponential backoff to prevent thundering herd
|
||||
- Preserved token format for backward compatibility
|
||||
|
||||
**Resolution:** Deployed Aug 31, zero 401s after 2 weeks monitoring.
|
||||
|
||||
---
|
||||
💾 Restore: bd compact --restore bd-42
|
||||
📊 Original: 2,341 bytes | Compressed: 468 bytes (80% reduction)
|
||||
|
||||
Total: 468 bytes (80% reduction)
|
||||
```
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Foundation (16 hours)
|
||||
1. Schema changes (compaction columns, snapshots table)
|
||||
2. Config keys
|
||||
3. Candidate identification queries
|
||||
4. Migration testing
|
||||
|
||||
### Phase 2: Core Compaction (20 hours)
|
||||
5. Haiku client with prompts
|
||||
6. Snapshot creation/restoration
|
||||
7. Tier 1 compaction logic
|
||||
8. CLI command (`bd compact`)
|
||||
|
||||
### Phase 3: Advanced Features (12 hours)
|
||||
9. Tier 2 compaction
|
||||
10. Restore functionality
|
||||
11. Stats command
|
||||
12. Event tracking
|
||||
|
||||
### Phase 4: Polish (12 hours)
|
||||
13. Comprehensive tests
|
||||
14. Documentation (README, COMPACTION.md)
|
||||
15. Examples and workflows
|
||||
|
||||
**Total MVP:** ~60 hours
|
||||
|
||||
### Optional Enhancements (Phase 5+)
|
||||
- Auto-compaction triggers
|
||||
- Git commit counting
|
||||
- Local model support (Ollama)
|
||||
- Custom prompt templates
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Issue (closed 45 days ago)
|
||||
↓
|
||||
Eligibility Check
|
||||
↓ (passes: all deps closed, >30 days)
|
||||
Create Snapshot
|
||||
↓
|
||||
Call Haiku API
|
||||
↓ (returns: structured summary)
|
||||
Update Issue
|
||||
↓
|
||||
Record Event
|
||||
↓
|
||||
Export to JSONL
|
||||
```
|
||||
|
||||
## Key Files
|
||||
|
||||
**New Files:**
|
||||
- `internal/compact/haiku.go` - Haiku API client
|
||||
- `internal/compact/compactor.go` - Core compaction logic
|
||||
- `internal/storage/sqlite/compact.go` - Candidate queries
|
||||
- `cmd/bd/compact.go` - CLI command
|
||||
|
||||
**Modified Files:**
|
||||
- `internal/storage/sqlite/schema.go` - Add snapshots table
|
||||
- `internal/storage/sqlite/sqlite.go` - Add migrations
|
||||
- `internal/types/types.go` - Add EventCompacted
|
||||
- `cmd/bd/show.go` - Display compaction status
|
||||
|
||||
## Schema Changes
|
||||
|
||||
```sql
|
||||
-- Add to issues table
|
||||
ALTER TABLE issues ADD COLUMN compaction_level INTEGER DEFAULT 0;
|
||||
ALTER TABLE issues ADD COLUMN compacted_at DATETIME;
|
||||
ALTER TABLE issues ADD COLUMN original_size INTEGER;
|
||||
|
||||
-- New table
|
||||
CREATE TABLE issue_snapshots (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
issue_id TEXT NOT NULL,
|
||||
snapshot_time DATETIME NOT NULL,
|
||||
compaction_level INTEGER NOT NULL,
|
||||
original_size INTEGER NOT NULL,
|
||||
compressed_size INTEGER NOT NULL,
|
||||
original_content TEXT NOT NULL, -- JSON
|
||||
archived_events TEXT,
|
||||
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
||||
);
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
```sql
|
||||
INSERT INTO config (key, value) VALUES
|
||||
('compact_tier1_days', '30'),
|
||||
('compact_tier1_dep_levels', '2'),
|
||||
('compact_tier2_days', '90'),
|
||||
('compact_tier2_dep_levels', '5'),
|
||||
('compact_tier2_commits', '100'),
|
||||
('compact_model', 'claude-3-5-haiku-20241022'),
|
||||
('compact_batch_size', '50'),
|
||||
('compact_parallel_workers', '5'),
|
||||
('auto_compact_enabled', 'false');
|
||||
```
|
||||
|
||||
## Haiku Prompts
|
||||
|
||||
### Tier 1 (300 words)
|
||||
```
|
||||
Summarize this closed software issue. Preserve key decisions,
|
||||
implementation approach, and outcome. Max 300 words.
|
||||
|
||||
Title: {{.Title}}
|
||||
Type: {{.IssueType}}
|
||||
Priority: {{.Priority}}
|
||||
|
||||
Description: {{.Description}}
|
||||
Design Notes: {{.Design}}
|
||||
Implementation Notes: {{.Notes}}
|
||||
Acceptance Criteria: {{.AcceptanceCriteria}}
|
||||
|
||||
Output format:
|
||||
**Summary:** [2-3 sentences: problem, solution, outcome]
|
||||
**Key Decisions:** [bullet points of non-obvious choices]
|
||||
**Resolution:** [how it was closed]
|
||||
```
|
||||
|
||||
### Tier 2 (150 words)
|
||||
```
|
||||
Ultra-compress this old closed issue to ≤150 words.
|
||||
Focus on lasting architectural impact.
|
||||
|
||||
Title: {{.Title}}
|
||||
Original Summary: {{.Description}}
|
||||
|
||||
Output a single paragraph covering:
|
||||
- What was built/fixed
|
||||
- Why it mattered
|
||||
- Lasting impact (if any)
|
||||
```
|
||||
|
||||
## Success Metrics
|
||||
|
||||
- **Space:** 70-85% reduction (Tier 1), 90-95% (Tier 2)
|
||||
- **Quality:** Summaries preserve essential context
|
||||
- **Safety:** 100% restore success rate
|
||||
- **Performance:** <100ms candidate query, ~2s per issue (Haiku)
|
||||
- **Cost:** <$1.50 per 1,000 issues
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Info loss in summaries | Dry-run review, restore capability, prompt tuning |
|
||||
| API rate limits | Exponential backoff, respect limits, batch sizing |
|
||||
| Accidental compression | Dry-run required, snapshots permanent |
|
||||
| JSONL conflicts | Compaction is deterministic, git handles well |
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Review design:** `COMPACTION_DESIGN.md` (comprehensive spec)
|
||||
2. **File issues:** `bd create -f compaction-issues.md` (16 issues)
|
||||
3. **Start implementation:** Begin with schema + config (Issues 1-2)
|
||||
4. **Iterate:** Build MVP (Tier 1 only), then enhance
|
||||
|
||||
## Questions?
|
||||
|
||||
- Full design: `COMPACTION_DESIGN.md`
|
||||
- Issue breakdown: `compaction-issues.md`
|
||||
- Reference implementation in Phase 1-4 above
|
||||
|
||||
---
|
||||
|
||||
**Ready to implement in another session!**
|
||||
779
compaction-issues.md
Normal file
779
compaction-issues.md
Normal file
@@ -0,0 +1,779 @@
|
||||
# Compaction Feature Issues
|
||||
|
||||
This file contains all issues for the database compaction feature, ready to import with:
|
||||
```bash
|
||||
bd create -f compaction-issues.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Epic: Add intelligent database compaction with Claude Haiku
|
||||
|
||||
### Type
|
||||
epic
|
||||
|
||||
### Priority
|
||||
2
|
||||
|
||||
### Description
|
||||
|
||||
Implement multi-tier database compaction using Claude Haiku to semantically compress old, closed issues. This keeps the database lightweight and agent-friendly while preserving essential context.
|
||||
|
||||
Goals:
|
||||
- 70-95% space reduction for eligible issues
|
||||
- Full restore capability via snapshots
|
||||
- Opt-in with dry-run safety
|
||||
- ~$1 per 1,000 issues compacted
|
||||
|
||||
### Acceptance Criteria
|
||||
- Schema migration with snapshots table
|
||||
- Haiku integration for summarization
|
||||
- Two-tier compaction (30d, 90d)
|
||||
- CLI with dry-run, restore, stats
|
||||
- Full test coverage
|
||||
- Documentation complete
|
||||
|
||||
### Labels
|
||||
compaction, epic, haiku, v1.1
|
||||
|
||||
---
|
||||
|
||||
## Add compaction schema and migrations
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
1
|
||||
|
||||
### Description
|
||||
|
||||
Add database schema support for issue compaction tracking and snapshot storage.
|
||||
|
||||
### Design
|
||||
|
||||
Add three columns to `issues` table:
|
||||
- `compaction_level INTEGER DEFAULT 0` - 0=original, 1=tier1, 2=tier2
|
||||
- `compacted_at DATETIME` - when last compacted
|
||||
- `original_size INTEGER` - bytes before first compaction
|
||||
|
||||
Create `issue_snapshots` table:
|
||||
```sql
|
||||
CREATE TABLE issue_snapshots (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
issue_id TEXT NOT NULL,
|
||||
snapshot_time DATETIME NOT NULL,
|
||||
compaction_level INTEGER NOT NULL,
|
||||
original_size INTEGER NOT NULL,
|
||||
compressed_size INTEGER NOT NULL,
|
||||
original_content TEXT NOT NULL, -- JSON blob
|
||||
archived_events TEXT,
|
||||
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
||||
);
|
||||
```
|
||||
|
||||
Add indexes:
|
||||
- `idx_snapshots_issue` on `issue_id`
|
||||
- `idx_snapshots_level` on `compaction_level`
|
||||
|
||||
Add migration functions in `internal/storage/sqlite/sqlite.go`:
|
||||
- `migrateCompactionColumns(db *sql.DB) error`
|
||||
- `migrateSnapshotsTable(db *sql.DB) error`
|
||||
|
||||
### Acceptance Criteria
|
||||
- Existing databases migrate automatically
|
||||
- New databases include columns by default
|
||||
- Migration is idempotent (safe to run multiple times)
|
||||
- No data loss during migration
|
||||
- Tests verify migration on fresh and existing DBs
|
||||
|
||||
### Labels
|
||||
compaction, schema, migration, database
|
||||
|
||||
---
|
||||
|
||||
## Add compaction configuration keys
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
1
|
||||
|
||||
### Description
|
||||
|
||||
Add configuration keys for compaction behavior with sensible defaults.
|
||||
|
||||
### Design
|
||||
|
||||
Add to `internal/storage/sqlite/schema.go` initial config:
|
||||
```sql
|
||||
INSERT OR IGNORE INTO config (key, value) VALUES
|
||||
('compact_tier1_days', '30'),
|
||||
('compact_tier1_dep_levels', '2'),
|
||||
('compact_tier2_days', '90'),
|
||||
('compact_tier2_dep_levels', '5'),
|
||||
('compact_tier2_commits', '100'),
|
||||
('compact_model', 'claude-3-5-haiku-20241022'),
|
||||
('compact_batch_size', '50'),
|
||||
('compact_parallel_workers', '5'),
|
||||
('auto_compact_enabled', 'false');
|
||||
```
|
||||
|
||||
Add helper functions for loading config into typed struct.
|
||||
|
||||
### Acceptance Criteria
|
||||
- Config keys created on init
|
||||
- Existing DBs get defaults on migration
|
||||
- `bd config get/set` works with all keys
|
||||
- Type validation (days=int, enabled=bool)
|
||||
- Documentation in README.md
|
||||
|
||||
### Labels
|
||||
compaction, config, configuration
|
||||
|
||||
---
|
||||
|
||||
## Implement candidate identification queries
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
1
|
||||
|
||||
### Description
|
||||
|
||||
Write SQL queries to identify issues eligible for Tier 1 and Tier 2 compaction based on closure time and dependency status.
|
||||
|
||||
### Design
|
||||
|
||||
Create `internal/storage/sqlite/compact.go` with:
|
||||
|
||||
```go
|
||||
type CompactionCandidate struct {
|
||||
IssueID string
|
||||
ClosedAt time.Time
|
||||
OriginalSize int
|
||||
EstimatedSize int
|
||||
DependentCount int
|
||||
}
|
||||
|
||||
func (s *SQLiteStorage) GetTier1Candidates(ctx context.Context) ([]*CompactionCandidate, error)
|
||||
func (s *SQLiteStorage) GetTier2Candidates(ctx context.Context) ([]*CompactionCandidate, error)
|
||||
func (s *SQLiteStorage) CheckEligibility(ctx context.Context, issueID string, tier int) (bool, string, error)
|
||||
```
|
||||
|
||||
Use recursive CTE for dependency depth checking (similar to ready_issues view).
|
||||
|
||||
### Acceptance Criteria
|
||||
- Tier 1 query filters by days and dependency depth
|
||||
- Tier 2 query includes commit/issue count checks
|
||||
- Dependency checking handles circular deps gracefully
|
||||
- Performance: <100ms for 10,000 issue database
|
||||
- Tests cover edge cases (no deps, circular deps, mixed status)
|
||||
|
||||
### Labels
|
||||
compaction, sql, query, dependencies
|
||||
|
||||
---
|
||||
|
||||
## Create Haiku client and prompt templates
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
1
|
||||
|
||||
### Description
|
||||
|
||||
Implement Claude Haiku API client with template-based prompts for Tier 1 and Tier 2 summarization.
|
||||
|
||||
### Design
|
||||
|
||||
Create `internal/compact/haiku.go`:
|
||||
|
||||
```go
|
||||
type HaikuClient struct {
|
||||
client *anthropic.Client
|
||||
model string
|
||||
}
|
||||
|
||||
func NewHaikuClient(apiKey string) (*HaikuClient, error)
|
||||
func (h *HaikuClient) SummarizeTier1(ctx context.Context, issue *types.Issue) (string, error)
|
||||
func (h *HaikuClient) SummarizeTier2(ctx context.Context, issue *types.Issue) (string, error)
|
||||
```
|
||||
|
||||
Use text/template for prompt rendering.
|
||||
|
||||
Tier 1 output format:
|
||||
```
|
||||
**Summary:** [2-3 sentences]
|
||||
**Key Decisions:** [bullet points]
|
||||
**Resolution:** [outcome]
|
||||
```
|
||||
|
||||
Tier 2 output format:
|
||||
```
|
||||
Single paragraph ≤150 words covering what was built, why it mattered, lasting impact.
|
||||
```
|
||||
|
||||
### Acceptance Criteria
|
||||
- API key from env var or config (env takes precedence)
|
||||
- Prompts render correctly with templates
|
||||
- Rate limiting handled gracefully (exponential backoff)
|
||||
- Network errors retry up to 3 times
|
||||
- Mock tests for API calls
|
||||
|
||||
### Labels
|
||||
compaction, haiku, api, llm
|
||||
|
||||
---
|
||||
|
||||
## Implement snapshot creation and restoration
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
1
|
||||
|
||||
### Description
|
||||
|
||||
Implement snapshot creation before compaction and restoration capability to undo compaction.
|
||||
|
||||
### Design
|
||||
|
||||
Add to `internal/storage/sqlite/compact.go`:
|
||||
|
||||
```go
|
||||
func (s *SQLiteStorage) CreateSnapshot(ctx context.Context, issue *types.Issue, level int) error
|
||||
func (s *SQLiteStorage) RestoreFromSnapshot(ctx context.Context, issueID string, level int) error
|
||||
func (s *SQLiteStorage) GetSnapshots(ctx context.Context, issueID string) ([]*Snapshot, error)
|
||||
```
|
||||
|
||||
Snapshot JSON structure:
|
||||
```json
|
||||
{
|
||||
"description": "...",
|
||||
"design": "...",
|
||||
"notes": "...",
|
||||
"acceptance_criteria": "...",
|
||||
"title": "..."
|
||||
}
|
||||
```
|
||||
|
||||
### Acceptance Criteria
|
||||
- Snapshot created atomically with compaction
|
||||
- Restore returns exact original content
|
||||
- Multiple snapshots per issue supported (Tier 1 → Tier 2)
|
||||
- JSON encoding handles UTF-8 and special characters
|
||||
- Size calculation is accurate (UTF-8 bytes)
|
||||
|
||||
### Labels
|
||||
compaction, snapshot, restore, safety
|
||||
|
||||
---
|
||||
|
||||
## Implement Tier 1 compaction logic
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
1
|
||||
|
||||
### Description
|
||||
|
||||
Implement the core Tier 1 compaction process: snapshot → summarize → update.
|
||||
|
||||
### Design
|
||||
|
||||
Add to `internal/compact/compactor.go`:
|
||||
|
||||
```go
|
||||
type Compactor struct {
|
||||
store storage.Storage
|
||||
haiku *HaikuClient
|
||||
config *CompactConfig
|
||||
}
|
||||
|
||||
func New(store storage.Storage, apiKey string, config *CompactConfig) (*Compactor, error)
|
||||
func (c *Compactor) CompactTier1(ctx context.Context, issueID string) error
|
||||
func (c *Compactor) CompactTier1Batch(ctx context.Context, issueIDs []string) error
|
||||
```
|
||||
|
||||
Process:
|
||||
1. Verify eligibility
|
||||
2. Calculate original size
|
||||
3. Create snapshot
|
||||
4. Call Haiku for summary
|
||||
5. Update issue (description=summary, clear design/notes/criteria)
|
||||
6. Set compaction_level=1, compacted_at=now, original_size
|
||||
7. Record EventCompacted
|
||||
8. Mark dirty for export
|
||||
|
||||
### Acceptance Criteria
|
||||
- Single issue compaction works end-to-end
|
||||
- Batch processing with parallel workers (5 concurrent)
|
||||
- Errors don't corrupt database (transaction rollback)
|
||||
- EventCompacted includes size savings
|
||||
- Dry-run mode (identify + size estimate only, no API calls)
|
||||
|
||||
### Labels
|
||||
compaction, tier1, core-logic
|
||||
|
||||
---
|
||||
|
||||
## Implement Tier 2 compaction logic
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
2
|
||||
|
||||
### Description
|
||||
|
||||
Implement Tier 2 ultra-compression: more aggressive summarization and optional event pruning.
|
||||
|
||||
### Design
|
||||
|
||||
Add to `internal/compact/compactor.go`:
|
||||
|
||||
```go
|
||||
func (c *Compactor) CompactTier2(ctx context.Context, issueID string) error
|
||||
func (c *Compactor) CompactTier2Batch(ctx context.Context, issueIDs []string) error
|
||||
```
|
||||
|
||||
Process:
|
||||
1. Verify issue is at compaction_level = 1
|
||||
2. Check Tier 2 eligibility (days, deps, commits/issues)
|
||||
3. Create Tier 2 snapshot
|
||||
4. Call Haiku with ultra-compression prompt
|
||||
5. Update issue (description = single paragraph, clear all other fields)
|
||||
6. Set compaction_level = 2
|
||||
7. Optionally prune events (keep created/closed, archive rest to snapshot)
|
||||
|
||||
### Acceptance Criteria
|
||||
- Requires existing Tier 1 compaction
|
||||
- Git commit counting works (with fallback to issue counter)
|
||||
- Events optionally pruned (config: compact_events_enabled)
|
||||
- Archived events stored in snapshot JSON
|
||||
- Size reduction 90-95%
|
||||
|
||||
### Labels
|
||||
compaction, tier2, advanced
|
||||
|
||||
---
|
||||
|
||||
## Add `bd compact` CLI command
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
1
|
||||
|
||||
### Description
|
||||
|
||||
Implement the `bd compact` command with dry-run, batch processing, and progress reporting.
|
||||
|
||||
### Design
|
||||
|
||||
Create `cmd/bd/compact.go`:
|
||||
|
||||
```go
|
||||
var compactCmd = &cobra.Command{
|
||||
Use: "compact",
|
||||
Short: "Compact old closed issues to save space",
|
||||
}
|
||||
|
||||
Flags:
|
||||
--dry-run Preview without compacting
|
||||
--tier int Compaction tier (1 or 2, default: 1)
|
||||
--all Process all candidates
|
||||
--id string Compact specific issue
|
||||
--force Force compact (bypass checks, requires --id)
|
||||
--batch-size int Issues per batch
|
||||
--workers int Parallel workers
|
||||
--json JSON output
|
||||
```
|
||||
|
||||
### Acceptance Criteria
|
||||
- `--dry-run` shows accurate preview with size estimates
|
||||
- `--all` processes all candidates
|
||||
- `--id` compacts single issue
|
||||
- `--force` bypasses eligibility checks (only with --id)
|
||||
- Progress bar for batches (e.g., [████████] 47/47)
|
||||
- JSON output with `--json`
|
||||
- Exit codes: 0=success, 1=error
|
||||
- Shows summary: count, size saved, cost, time
|
||||
|
||||
### Labels
|
||||
compaction, cli, command
|
||||
|
||||
---
|
||||
|
||||
## Add `bd compact --restore` functionality
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
2
|
||||
|
||||
### Description
|
||||
|
||||
Implement restore command to undo compaction from snapshots.
|
||||
|
||||
### Design
|
||||
|
||||
Add to `cmd/bd/compact.go`:
|
||||
|
||||
```go
|
||||
var compactRestore string
|
||||
|
||||
compactCmd.Flags().StringVar(&compactRestore, "restore", "", "Restore issue from snapshot")
|
||||
```
|
||||
|
||||
Process:
|
||||
1. Load snapshot for issue
|
||||
2. Parse JSON content
|
||||
3. Update issue with original content
|
||||
4. Set compaction_level = 0, compacted_at = NULL, original_size = NULL
|
||||
5. Record event (EventRestored or EventUpdated)
|
||||
6. Mark dirty for export
|
||||
|
||||
### Acceptance Criteria
|
||||
- Restores exact original content
|
||||
- Handles multiple snapshots (use latest by default)
|
||||
- `--level` flag to choose specific snapshot
|
||||
- Updates compaction_level correctly
|
||||
- Exports restored content to JSONL
|
||||
- Shows before/after in output
|
||||
|
||||
### Labels
|
||||
compaction, restore, cli
|
||||
|
||||
---
|
||||
|
||||
## Add `bd compact --stats` command
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
2
|
||||
|
||||
### Description
|
||||
|
||||
Add statistics command showing compaction status and potential savings.
|
||||
|
||||
### Design
|
||||
|
||||
```go
|
||||
var compactStats bool
|
||||
|
||||
compactCmd.Flags().BoolVar(&compactStats, "stats", false, "Show compaction statistics")
|
||||
```
|
||||
|
||||
Output:
|
||||
- Total issues, by compaction level (0, 1, 2)
|
||||
- Current DB size vs estimated uncompacted size
|
||||
- Space savings (KB/MB and %)
|
||||
- Candidates for each tier with size estimates
|
||||
- Estimated API cost (Haiku pricing)
|
||||
|
||||
### Acceptance Criteria
|
||||
- Accurate counts by compaction_level
|
||||
- Size calculations include all text fields (UTF-8 bytes)
|
||||
- Shows candidates with eligibility reasons
|
||||
- Cost estimation based on current Haiku pricing
|
||||
- JSON output supported
|
||||
- Clear, readable table format
|
||||
|
||||
### Labels
|
||||
compaction, stats, reporting
|
||||
|
||||
---
|
||||
|
||||
## Add EventCompacted to event system
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
2
|
||||
|
||||
### Description
|
||||
|
||||
Add new event type for tracking compaction in audit trail.
|
||||
|
||||
### Design
|
||||
|
||||
1. Add to `internal/types/types.go`:
|
||||
```go
|
||||
const EventCompacted EventType = "compacted"
|
||||
```
|
||||
|
||||
2. Record event during compaction:
|
||||
```go
|
||||
eventData := map[string]interface{}{
|
||||
"tier": tier,
|
||||
"original_size": originalSize,
|
||||
"compressed_size": compressedSize,
|
||||
"reduction_pct": (1 - float64(compressedSize)/float64(originalSize)) * 100,
|
||||
}
|
||||
```
|
||||
|
||||
3. Update event display in `bd show`.
|
||||
|
||||
### Acceptance Criteria
|
||||
- Event includes tier, original_size, compressed_size, reduction_pct
|
||||
- Shows in event history (`bd events <id>`)
|
||||
- Exports to JSONL correctly
|
||||
- `bd show` displays compaction status and marker
|
||||
|
||||
### Labels
|
||||
compaction, events, audit
|
||||
|
||||
---
|
||||
|
||||
## Add compaction indicator to `bd show`
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
2
|
||||
|
||||
### Description
|
||||
|
||||
Update `bd show` command to display compaction status prominently.
|
||||
|
||||
### Design
|
||||
|
||||
Add to issue display:
|
||||
```
|
||||
bd-42: Fix authentication bug [CLOSED] 🗜️
|
||||
|
||||
Status: closed (compacted L1)
|
||||
...
|
||||
|
||||
---
|
||||
💾 Restore: bd compact --restore bd-42
|
||||
📊 Original: 2,341 bytes | Compressed: 468 bytes (80% reduction)
|
||||
🗜️ Compacted: 2025-10-15 (Tier 1)
|
||||
```
|
||||
|
||||
Emoji indicators:
|
||||
- Tier 1: 🗜️
|
||||
- Tier 2: 📦
|
||||
|
||||
### Acceptance Criteria
|
||||
- Compaction status visible in title line
|
||||
- Footer shows size savings when compacted
|
||||
- Restore command shown for compacted issues
|
||||
- Works with `--json` output (includes compaction fields)
|
||||
- Emoji optional (controlled by config or terminal detection)
|
||||
|
||||
### Labels
|
||||
compaction, ui, display
|
||||
|
||||
---
|
||||
|
||||
## Write compaction tests
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
1
|
||||
|
||||
### Description
|
||||
|
||||
Comprehensive test suite for compaction functionality.
|
||||
|
||||
### Design
|
||||
|
||||
Test coverage:
|
||||
|
||||
1. **Candidate Identification:**
|
||||
- Eligibility by time
|
||||
- Dependency depth checking
|
||||
- Mixed status dependents
|
||||
- Edge cases (no deps, circular deps)
|
||||
|
||||
2. **Snapshots:**
|
||||
- Create and restore
|
||||
- Multiple snapshots per issue
|
||||
- Content integrity (UTF-8, special chars)
|
||||
|
||||
3. **Tier 1 Compaction:**
|
||||
- Single issue compaction
|
||||
- Batch processing
|
||||
- Error handling (API failures)
|
||||
|
||||
4. **Tier 2 Compaction:**
|
||||
- Requires Tier 1
|
||||
- Events pruning
|
||||
- Commit counting fallback
|
||||
|
||||
5. **CLI:**
|
||||
- All flag combinations
|
||||
- Dry-run accuracy
|
||||
- JSON output parsing
|
||||
|
||||
6. **Integration:**
|
||||
- End-to-end flow
|
||||
- JSONL export/import
|
||||
- Restore verification
|
||||
|
||||
### Acceptance Criteria
|
||||
- Test coverage >80%
|
||||
- All edge cases covered
|
||||
- Mock Haiku API in tests (no real API calls)
|
||||
- Integration tests pass
|
||||
- `go test ./...` passes
|
||||
- Benchmarks for performance-critical paths
|
||||
|
||||
### Labels
|
||||
compaction, testing, quality
|
||||
|
||||
---
|
||||
|
||||
## Add compaction documentation
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
2
|
||||
|
||||
### Description
|
||||
|
||||
Document compaction feature in README and create detailed COMPACTION.md guide.
|
||||
|
||||
### Design
|
||||
|
||||
**Update README.md:**
|
||||
- Add to Features section
|
||||
- CLI examples (dry-run, compact, restore, stats)
|
||||
- Configuration guide
|
||||
- Cost analysis
|
||||
|
||||
**Create COMPACTION.md:**
|
||||
- How compaction works (architecture overview)
|
||||
- When to use each tier
|
||||
- Detailed cost analysis with examples
|
||||
- Safety mechanisms (snapshots, restore, dry-run)
|
||||
- Troubleshooting guide
|
||||
- FAQ
|
||||
|
||||
**Create examples/compaction/:**
|
||||
- `workflow.sh` - Example monthly compaction workflow
|
||||
- `cron-compact.sh` - Cron job setup
|
||||
- `auto-compact.sh` - Auto-compaction script
|
||||
|
||||
### Acceptance Criteria
|
||||
- README.md updated with compaction section
|
||||
- COMPACTION.md comprehensive and clear
|
||||
- Examples work as documented (tested)
|
||||
- Screenshots or ASCII examples included
|
||||
- API key setup documented (env var vs config)
|
||||
- Covers common questions and issues
|
||||
|
||||
### Labels
|
||||
compaction, docs, documentation, examples
|
||||
|
||||
---
|
||||
|
||||
## Optional: Implement auto-compaction
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
3
|
||||
|
||||
### Description
|
||||
|
||||
Implement automatic compaction triggered by certain operations when enabled via config.
|
||||
|
||||
### Design
|
||||
|
||||
Trigger points (when `auto_compact_enabled = true`):
|
||||
1. `bd stats` - check and compact if candidates exist
|
||||
2. `bd export` - before exporting
|
||||
3. Configurable: on any read operation after N candidates accumulate
|
||||
|
||||
Add:
|
||||
```go
|
||||
func (s *SQLiteStorage) AutoCompact(ctx context.Context) error {
|
||||
enabled, _ := s.GetConfig(ctx, "auto_compact_enabled")
|
||||
if enabled != "true" {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Run Tier 1 compaction on all candidates
|
||||
// Limit to batch_size to avoid long operations
|
||||
// Log activity for transparency
|
||||
}
|
||||
```
|
||||
|
||||
### Acceptance Criteria
|
||||
- Respects auto_compact_enabled config (default: false)
|
||||
- Limits batch size to avoid blocking operations
|
||||
- Logs compaction activity (visible with --verbose)
|
||||
- Can be disabled per-command with `--no-auto-compact` flag
|
||||
- Only compacts Tier 1 (Tier 2 remains manual)
|
||||
- Doesn't run more than once per hour (rate limiting)
|
||||
|
||||
### Labels
|
||||
compaction, automation, optional, v1.2
|
||||
|
||||
---
|
||||
|
||||
## Optional: Add git commit counting
|
||||
|
||||
### Type
|
||||
task
|
||||
|
||||
### Priority
|
||||
3
|
||||
|
||||
### Description
|
||||
|
||||
Implement git commit counting for "project time" measurement as alternative to calendar time for Tier 2 eligibility.
|
||||
|
||||
### Design
|
||||
|
||||
```go
|
||||
func getCommitsSince(closedAt time.Time) (int, error) {
|
||||
cmd := exec.Command("git", "rev-list", "--count",
|
||||
fmt.Sprintf("--since=%s", closedAt.Format(time.RFC3339)), "HEAD")
|
||||
output, err := cmd.Output()
|
||||
if err != nil {
|
||||
return 0, err // Not in git repo or git not available
|
||||
}
|
||||
return strconv.Atoi(strings.TrimSpace(string(output)))
|
||||
}
|
||||
```
|
||||
|
||||
Fallback strategies:
|
||||
1. Git commit count (preferred)
|
||||
2. Issue counter delta (store counter at close time, compare later)
|
||||
3. Pure time-based (90 days)
|
||||
|
||||
### Acceptance Criteria
|
||||
- Counts commits since closed_at timestamp
|
||||
- Handles git not available gracefully (falls back)
|
||||
- Fallback to issue counter delta works
|
||||
- Configurable via compact_tier2_commits config key
|
||||
- Tested with real git repo
|
||||
- Works in non-git environments
|
||||
|
||||
### Labels
|
||||
compaction, git, optional, tier2
|
||||
Reference in New Issue
Block a user