johno/beads
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix compaction docs: remove non-existent restore feature

Browse Source 
Compaction is permanent graceful decay - restore functionality
doesn't exist. Updated docs to reflect reality:
- Removed restore command examples
- Removed snapshot/restore sections
- Updated to explain git history recovery option
- Clarified permanent nature throughout
This commit is contained in:
Steve Yegge
2025-10-16 01:11:35 -07:00
parent 54469936a7
commit ebcc503849
3 changed files with 26 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.beads/issues.jsonl
View File

@@ -181,7 +181,7 @@
{"id":"bd-262","title":"Add EventCompacted to event system","description":"Add new event type for tracking compaction in audit trail.","design":"1. Add to `internal/types/types.go`:\n```go\nconst EventCompacted EventType = \"compacted\"\n```\n\n2. Record event during compaction:\n```go\neventData := map[string]interface{}{\n \"tier\": tier,\n \"original_size\": originalSize,\n \"compressed_size\": compressedSize,\n \"reduction_pct\": (1 - float64(compressedSize)/float64(originalSize)) * 100,\n}\n```\n\n3. Update event display in `bd show`.","acceptance_criteria":"- Event includes tier, original_size, compressed_size, reduction_pct\n- Shows in event history (`bd events \u003cid\u003e`)\n- Exports to JSONL correctly\n- `bd show` displays compaction status and marker","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-15T21:51:23.244219-07:00","updated_at":"2025-10-16T01:00:55.693486-07:00","closed_at":"2025-10-16T00:59:17.465182-07:00"}
{"id":"bd-263","title":"Add compaction indicator to `bd show`","description":"Update `bd show` command to display compaction status prominently.","design":"Add to issue display:\n```\nbd-42: Fix authentication bug [CLOSED] 🗜️\n\nStatus: closed (compacted L1)\n...\n\n---\n💾 Restore: bd compact --restore bd-42\n📊 Original: 2,341 bytes | Compressed: 468 bytes (80% reduction)\n🗜️ Compacted: 2025-10-15 (Tier 1)\n```\n\nEmoji indicators:\n- Tier 1: 🗜️\n- Tier 2: 📦","acceptance_criteria":"- Compaction status visible in title line\n- Footer shows size savings when compacted\n- Restore command shown for compacted issues\n- Works with `--json` output (includes compaction fields)\n- Emoji optional (controlled by config or terminal detection)","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-15T21:51:23.253091-07:00","updated_at":"2025-10-16T01:03:59.28912-07:00","closed_at":"2025-10-16T01:03:59.28912-07:00"}
{"id":"bd-264","title":"Write compaction tests","description":"Comprehensive test suite for compaction functionality.","design":"Test coverage:\n\n1. **Candidate Identification:**\n - Eligibility by time\n - Dependency depth checking\n - Mixed status dependents\n - Edge cases (no deps, circular deps)\n\n2. **Snapshots:**\n - Create and restore\n - Multiple snapshots per issue\n - Content integrity (UTF-8, special chars)\n\n3. **Tier 1 Compaction:**\n - Single issue compaction\n - Batch processing\n - Error handling (API failures)\n\n4. **Tier 2 Compaction:**\n - Requires Tier 1\n - Events pruning\n - Commit counting fallback\n\n5. **CLI:**\n - All flag combinations\n - Dry-run accuracy\n - JSON output parsing\n\n6. **Integration:**\n - End-to-end flow\n - JSONL export/import\n - Restore verification","acceptance_criteria":"- Test coverage \u003e80%\n- All edge cases covered\n- Mock Haiku API in tests (no real API calls)\n- Integration tests pass\n- `go test ./...` passes\n- Benchmarks for performance-critical paths","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-15T21:51:23.262504-07:00","updated_at":"2025-10-16T01:00:55.694731-07:00","closed_at":"2025-10-16T00:02:11.246331-07:00"}
{"id":"bd-265","title":"Add compaction documentation","description":"Document compaction feature in README and create detailed COMPACTION.md guide.","design":"**Update README.md:**\n- Add to Features section\n- CLI examples (dry-run, compact, restore, stats)\n- Configuration guide\n- Cost analysis\n\n**Create COMPACTION.md:**\n- How compaction works (architecture overview)\n- When to use each tier\n- Detailed cost analysis with examples\n- Safety mechanisms (snapshots, restore, dry-run)\n- Troubleshooting guide\n- FAQ\n\n**Create examples/compaction/:**\n- `workflow.sh` - Example monthly compaction workflow\n- `cron-compact.sh` - Cron job setup\n- `auto-compact.sh` - Auto-compaction script","acceptance_criteria":"- README.md updated with compaction section\n- COMPACTION.md comprehensive and clear\n- Examples work as documented (tested)\n- Screenshots or ASCII examples included\n- API key setup documented (env var vs config)\n- Covers common questions and issues","status":"in_progress","priority":2,"issue_type":"task","created_at":"2025-10-15T21:51:23.265589-07:00","updated_at":"2025-10-16T01:06:45.804764-07:00"}
{"id":"bd-265","title":"Add compaction documentation","description":"Document compaction feature in README and create detailed COMPACTION.md guide.","design":"**Update README.md:**\n- Add to Features section\n- CLI examples (dry-run, compact, restore, stats)\n- Configuration guide\n- Cost analysis\n\n**Create COMPACTION.md:**\n- How compaction works (architecture overview)\n- When to use each tier\n- Detailed cost analysis with examples\n- Safety mechanisms (snapshots, restore, dry-run)\n- Troubleshooting guide\n- FAQ\n\n**Create examples/compaction/:**\n- `workflow.sh` - Example monthly compaction workflow\n- `cron-compact.sh` - Cron job setup\n- `auto-compact.sh` - Auto-compaction script","acceptance_criteria":"- README.md updated with compaction section\n- COMPACTION.md comprehensive and clear\n- Examples work as documented (tested)\n- Screenshots or ASCII examples included\n- API key setup documented (env var vs config)\n- Covers common questions and issues","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-15T21:51:23.265589-07:00","updated_at":"2025-10-16T01:09:54.124394-07:00","closed_at":"2025-10-16T01:09:54.124394-07:00"}
{"id":"bd-266","title":"Optional: Implement auto-compaction","description":"Implement automatic compaction triggered by certain operations when enabled via config.","design":"Trigger points (when `auto_compact_enabled = true`):\n1. `bd stats` - check and compact if candidates exist\n2. `bd export` - before exporting\n3. Configurable: on any read operation after N candidates accumulate\n\nAdd:\n```go\nfunc (s *SQLiteStorage) AutoCompact(ctx context.Context) error {\n enabled, _ := s.GetConfig(ctx, \"auto_compact_enabled\")\n if enabled != \"true\" {\n return nil\n }\n\n // Run Tier 1 compaction on all candidates\n // Limit to batch_size to avoid long operations\n // Log activity for transparency\n}\n```","acceptance_criteria":"- Respects auto_compact_enabled config (default: false)\n- Limits batch size to avoid blocking operations\n- Logs compaction activity (visible with --verbose)\n- Can be disabled per-command with `--no-auto-compact` flag\n- Only compacts Tier 1 (Tier 2 remains manual)\n- Doesn't run more than once per hour (rate limiting)","status":"open","priority":3,"issue_type":"task","created_at":"2025-10-15T21:51:23.281006-07:00","updated_at":"2025-10-16T01:00:55.695589-07:00"}
{"id":"bd-267","title":"Optional: Add git commit counting","description":"Implement git commit counting for \"project time\" measurement as alternative to calendar time for Tier 2 eligibility.","design":"```go\nfunc getCommitsSince(closedAt time.Time) (int, error) {\n cmd := exec.Command(\"git\", \"rev-list\", \"--count\",\n fmt.Sprintf(\"--since=%s\", closedAt.Format(time.RFC3339)), \"HEAD\")\n output, err := cmd.Output()\n if err != nil {\n return 0, err // Not in git repo or git not available\n }\n return strconv.Atoi(strings.TrimSpace(string(output)))\n}\n```\n\nFallback strategies:\n1. Git commit count (preferred)\n2. Issue counter delta (store counter at close time, compare later)\n3. Pure time-based (90 days)","acceptance_criteria":"- Counts commits since closed_at timestamp\n- Handles git not available gracefully (falls back)\n- Fallback to issue counter delta works\n- Configurable via compact_tier2_commits config key\n- Tested with real git repo\n- Works in non-git environments","status":"open","priority":3,"issue_type":"task","created_at":"2025-10-15T21:51:23.284781-07:00","updated_at":"2025-10-16T01:00:55.695972-07:00"}
{"id":"bd-268","title":"Explore in-memory embedded SQL alternatives to SQLite","description":"Investigate lightweight in-memory embedded SQL databases as alternative backends for environments where SQLite is problematic or considered too heavyweight. This would provide flexibility for different deployment scenarios.","design":"Research options:\n- modernc.org/sqlite (pure Go SQLite implementation, no cgo)\n- rqlite (distributed SQLite with Raft)\n- go-memdb (in-memory database by HashiCorp)\n- badger (embedded key-value store, would need SQL layer)\n- bbolt (embedded key-value store)\n- duckdb (lightweight analytical database)\n\nEvaluate on:\n- Memory footprint vs SQLite\n- cgo dependency (pure Go preferred)\n- SQL compatibility level\n- Transaction support\n- Performance characteristics\n- Maintenance/community status\n- Migration complexity from SQLite\n\nConsider creating a storage abstraction layer to support multiple backends.","acceptance_criteria":"- Document comparison of at least 3 alternatives\n- Benchmark memory usage and performance vs SQLite\n- Assess migration effort for each option\n- Recommendation on whether to support alternatives\n- If yes, prototype storage interface abstraction","notes":"Worth noting: modernc.org/sqlite is a pure Go implementation (no cgo) that might already address the \"heavyweight\" concern, since much of SQLite's overhead comes from cgo calls. Should evaluate this first before exploring completely different database technologies.","status":"open","priority":3,"issue_type":"task","created_at":"2025-10-15T23:17:33.560045-07:00","updated_at":"2025-10-16T01:00:55.696359-07:00"}

78
COMPACTION.md
View File

@@ -8,8 +8,8 @@ Beads compaction is **agentic memory decay** - your database naturally forgets f
- **Semantic compression**: Claude Haiku summarizes issues intelligently, preserving decisions and outcomes
- **Two-tier system**: Gradual decay from full detail → summary → ultra-brief
- **Full recovery**: Snapshots enable complete restoration if needed
- **Safe by design**: Dry-run preview, eligibility checks, snapshot verification
- **Permanent decay**: Original content is discarded to save space (not reversible)
- **Safe by design**: Dry-run preview, eligibility checks, git history preserves old versions
## How It Works
@@ -19,10 +19,9 @@ Beads compaction is **agentic memory decay** - your database naturally forgets f
**Process**:
1. Check eligibility (closed, 30+ days, no blockers)
2. Create snapshot (full JSON backup)
3. Send to Claude Haiku for summarization
4. Replace verbose fields with concise summary
5. Store original size for statistics
2. Send to Claude Haiku for summarization
3. Replace verbose fields with concise summary
4. Store original size for statistics
**Result**: 70-80% space reduction
@@ -55,9 +54,8 @@ Resolution: Deployed in v1.2.3.
**Process**:
1. Verify existing Tier 1 compaction
2. Check reference frequency (git commits, other issues)
3. Create Tier 2 snapshot
4. Ultra-compress to single paragraph
5. Optionally prune events (keep created/closed only)
3. Ultra-compress to single paragraph
4. Optionally prune events (keep created/closed only)
**Result**: 90-95% space reduction
@@ -117,16 +115,6 @@ bd compact --stats
# Estimated cost: $0.85
```
### Restore from Snapshot
```bash
# Restore compacted issue to original state
bd compact --restore bd-42
# Show the issue to verify
bd show bd-42
```
## Eligibility Rules
### Tier 1 Eligibility
@@ -314,28 +302,19 @@ fi
## Safety & Recovery
### Snapshots
### Git History
Every compaction creates a snapshot in the `compaction_snapshots` table:
```sql
CREATE TABLE compaction_snapshots (
id INTEGER PRIMARY KEY,
issue_id TEXT NOT NULL,
tier INTEGER NOT NULL,
snapshot_data TEXT NOT NULL, -- Full JSON of original issue
created_at DATETIME NOT NULL
);
```
### Restore Process
Compaction is permanent - the original content is discarded to save space. However, you can recover old versions from git history:
```bash
# Restore single issue
bd compact --restore bd-42
# View issue before compaction
git log -p -- .beads/issues.jsonl | grep -A 50 "bd-42"
# Verify restoration
bd show bd-42 # Should show original content
# Checkout old version
git checkout <commit-hash> -- .beads/issues.jsonl
# Or use git show
git show <commit-hash>:.beads/issues.jsonl | grep -A 50 "bd-42"
```
### Verification
@@ -348,9 +327,6 @@ bd compact --stats
# Spot-check compacted issues
bd show bd-42
# Verify snapshots exist
sqlite3 issues.db "SELECT COUNT(*) FROM compaction_snapshots;"
```
## Troubleshooting
@@ -383,14 +359,6 @@ Force compact (if you know what you're doing):
bd compact --id bd-42 --force
```
### Restore Failed
Snapshots are stored in SQLite. If restore fails, manually query:
```bash
sqlite3 issues.db "SELECT snapshot_data FROM compaction_snapshots WHERE issue_id='bd-42' ORDER BY created_at DESC LIMIT 1;"
```
## FAQ
### When should I compact?
@@ -400,9 +368,12 @@ sqlite3 issues.db "SELECT snapshot_data FROM compaction_snapshots WHERE issue_id
- **Large projects (5000+ issues)**: Monthly or quarterly
- **High-velocity teams**: Set up automated monthly compaction
### Can I restore compacted issues?
### Can I recover compacted issues?
**Yes!** Full snapshots are stored. Use `bd compact --restore <id>` anytime.
Compaction is permanent, but you can recover from git history:
```bash
git log -p -- .beads/issues.jsonl | grep -A 50 "bd-42"
```
### What happens to dependencies?
@@ -424,7 +395,7 @@ git push
### What if my team disagrees on compaction frequency?
Use `bd compact --dry-run` to preview. Discuss the candidates before running. You can always restore if someone needs the original.
Use `bd compact --dry-run` to preview. Discuss the candidates before running. Since compaction is permanent, get team consensus first.
### Can I compact open issues?
@@ -452,9 +423,8 @@ Not yet, but it's planned (bd-264). The current prompt is optimized for preservi
2. **Compact regularly**: Monthly or quarterly depending on project size
3. **Monitor costs**: Use `bd compact --stats` to track savings
4. **Automate it**: Set up cron jobs for hands-off maintenance
5. **Check snapshots**: Periodically verify snapshots are being created
6. **Commit results**: Always commit and push after compaction
7. **Team communication**: Let team know before large compaction runs
5. **Commit results**: Always commit and push after compaction
6. **Team communication**: Let team know before large compaction runs (it's permanent!)
## Examples

5
README.md
View File

@@ -447,9 +447,6 @@ bd compact --id bd-42
# Force compact (bypass eligibility checks)
bd compact --id bd-42 --force
# Restore from snapshot (full recovery)
bd compact --restore bd-42
# Tier 2 ultra-compression (90+ days, 95% reduction)
bd compact --tier 2 --all
```
@@ -467,7 +464,7 @@ Compaction uses Claude Haiku to semantically summarize issues:
- Tier 1: 30+ days since closed, no open dependents
- Tier 2: 90+ days since closed, rarely referenced in commits/issues
**Safety:** Full snapshots are kept - you can restore any compacted issue to its original state.
**Note:** Compaction is permanent graceful decay - original content is discarded to save space. Use git history to recover old versions if needed.
See [COMPACTION.md](COMPACTION.md) for detailed documentation, cost analysis, and automation examples.