From a0f4a9cacd9a7b0c6722035769adf113ddd4d4f4 Mon Sep 17 00:00:00 2001 From: Steve Yegge Date: Sat, 13 Dec 2025 10:16:27 -0800 Subject: [PATCH] feat(doctor): add tombstone health checks (bd-s3v) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add two new doctor checks for tombstone health: 1. Updated Deletions Manifest check: - Warns when legacy deletions.jsonl has entries (suggests migration) - Shows "Migrated to tombstones" when .migrated file exists - Shows "Using inline tombstones" for new repos 2. New Tombstones check: - Reports total tombstone count - Warns about expired tombstones (older than 30 days) - Shows tombstones expiring within 7 days - Suggests 'bd compact' to prune expired tombstones 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- .beads/issues.jsonl | 3 +- cmd/bd/doctor.go | 150 +++++++++++++++++++++++++++++++++++++------- 2 files changed, 130 insertions(+), 23 deletions(-) diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 9d927205..1fa589ba 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -386,7 +386,6 @@ {"id":"bd-hm8","title":"Add uninstall documentation (GH #445)","description":"","status":"closed","priority":2,"issue_type":"task","created_at":"2025-12-01T21:04:30.313637-08:00","updated_at":"2025-12-01T21:04:57.943192-08:00","closed_at":"2025-12-01T21:04:57.943192-08:00"} {"id":"bd-ho5","title":"Add 'town report' command for aggregated swarm status","description":"## Problem\nGetting a full swarm status requires running 6+ commands:\n- `town list \u003crig\u003e` for each rig\n- `town mail inbox` as Boss\n- `bd list --status=open/in_progress` per rig\n\nThis is slow and error-prone for both humans and agents.\n\n## Proposed Solution\nAdd `town report [RIG]` command that aggregates:\n- All rigs with polecat states (running/stopped, awake/asleep)\n- Boss inbox summary (unread count, recent senders)\n- Aggregate issue counts per rig (open/in_progress/blocked)\n\nExample output:\n```\n=== beads ===\nPolecats: 5 (5 running, 0 stopped)\nIssues: 20 open, 0 in_progress, 0 blocked\n\n=== gastown ===\nPolecats: 6 (4 running, 2 stopped)\nIssues: 0 open, 0 in_progress, 0 blocked\n\n=== Boss Mail ===\nUnread: 10 | Total: 22\nRecent: rictus (21:19), scrotus (21:14), immortanjoe (21:14)\n```\n\n## Acceptance Criteria\n- [ ] `town report` shows all rigs\n- [ ] `town report \u003crig\u003e` shows single rig detail\n- [ ] Output is concise and scannable\n- [ ] Completes in \u003c2 seconds","status":"closed","priority":2,"issue_type":"feature","created_at":"2025-11-27T22:55:36.8919-08:00","updated_at":"2025-11-27T22:56:08.071838-08:00","closed_at":"2025-11-27T22:56:08.071838-08:00"} {"id":"bd-hp0m","title":"Add test for legacy deletions.jsonl to tombstone conversion","description":"The importer now converts legacy deletions.jsonl entries to tombstones (bd-dve), but there's no dedicated test that:\n1. Creates a deletions.jsonl with entries\n2. Imports issues (some in deletions, some not)\n3. Verifies the converted tombstones have correct fields from the deletion record\n\nThe existing TestImportIssues_TombstoneNotFilteredByDeletionsManifest tests that tombstones bypass the filter, but doesn't verify the conversion of legacy deletions to tombstones.\n\nAdd a test: TestImportIssues_LegacyDeletionsConvertedToTombstones","status":"closed","priority":2,"issue_type":"task","created_at":"2025-12-07T01:41:12.976726-08:00","updated_at":"2025-12-07T02:16:16.882286-08:00","closed_at":"2025-12-07T02:16:16.882286-08:00","dependencies":[{"issue_id":"bd-hp0m","depends_on_id":"bd-dve","type":"blocks","created_at":"2025-12-07T01:41:28.391366-08:00","created_by":"daemon"}]} -{"id":"bd-hxou","title":"Daemon performAutoImport should update jsonl_content_hash after import","description":"The daemon's performAutoImport function was not updating jsonl_content_hash after successful import, unlike the CLI import path. This could cause repeated imports if the file hash and DB hash diverge.\n\nFix: Added hash update after validatePostImport succeeds, using repoKey for multi-repo support.","status":"closed","priority":2,"issue_type":"bug","created_at":"2025-12-13T06:44:04.442976-08:00","updated_at":"2025-12-13T06:44:09.546976-08:00","closed_at":"2025-12-13T06:44:09.546976-08:00"} {"id":"bd-ig5","title":"Duplicate tombstone constants in merge and types packages","description":"## Problem\n\nThe tombstone constants are duplicated between two packages:\n\n**internal/types/types.go:**\n```go\nconst DefaultTombstoneTTL = 30 * 24 * time.Hour\nconst ClockSkewGrace = 1 * time.Hour\nconst StatusTombstone Status = \"tombstone\"\n```\n\n**internal/merge/merge.go:**\n```go\nconst StatusTombstone = \"tombstone\"\nconst DefaultTombstoneTTL = 30 * 24 * time.Hour\nconst ClockSkewGrace = 1 * time.Hour\n```\n\nThis violates DRY and creates risk of divergence if one is updated but not the other.\n\n## Root Cause\n\nThe merge package has its own `Issue` struct (for JSONL parsing) and cannot import types.Issue directly due to the different struct design. However, the constants could be shared.\n\n## Recommendation\n\nExport the constants from types package and import them in merge:\n\n```go\n// merge/merge.go\nimport \"github.com/steveyegge/beads/internal/types\"\n\n// Use types.StatusTombstone, types.DefaultTombstoneTTL, types.ClockSkewGrace\n```\n\nOr create a shared constants package if import cycles are a concern.\n\n## Files\n- internal/merge/merge.go:241-248\n- internal/types/types.go:77-84, 170","status":"closed","priority":2,"issue_type":"bug","created_at":"2025-12-05T16:36:06.159443-08:00","updated_at":"2025-12-05T17:15:57.624376-08:00","closed_at":"2025-12-05T17:15:57.624376-08:00"} {"id":"bd-iip5","title":"TestImportIssues_LegacyDeletionsConvertedToTombstones is failing","description":"The test TestImportIssues_LegacyDeletionsConvertedToTombstones in internal/importer/importer_test.go is failing:\n\nExpected 3 created (1 regular + 2 tombstones from deletions.jsonl), got 2\nExpected tombstone for test-deleted2 not found\n\nThe test expects legacy deletions.jsonl entries to be converted to tombstones during import, but test-deleted2 is not being converted.\n\nLocation: internal/importer/importer_test.go:1344","status":"closed","priority":2,"issue_type":"bug","created_at":"2025-12-07T02:09:19.17774-08:00","updated_at":"2025-12-07T02:16:55.422522-08:00","closed_at":"2025-12-07T02:16:55.422522-08:00"} {"id":"bd-imj","title":"Deletion propagation via deletions manifest","description":"## Problem\n\nWhen `bd cleanup -f` or `bd delete` removes issues in one clone, those deletions don't propagate to other clones. The import logic only creates/updates, never deletes. This causes \"resurrection\" where deleted issues reappear.\n\n## Root Cause\n\nImport sees DB issues not in JSONL and assumes they're \"local unpushed work\" rather than \"intentionally deleted upstream.\"\n\n## Solution: Deletions Manifest\n\nAdd `.beads/deletions.jsonl` - an append-only log of deleted issue IDs with metadata.\n\n### Format\n```jsonl\n{\"id\":\"bd-xxx\",\"ts\":\"2025-11-25T10:00:00Z\",\"by\":\"stevey\"}\n{\"id\":\"bd-yyy\",\"ts\":\"2025-11-25T10:05:00Z\",\"by\":\"claude\",\"reason\":\"duplicate of bd-zzz\"}\n```\n\n### Fields\n- `id`: Issue ID (required)\n- `ts`: ISO 8601 UTC timestamp (required)\n- `by`: Actor who deleted (required)\n- `reason`: Optional context (\"cleanup\", \"duplicate of X\", etc.)\n\n### Import Logic\n```\nFor each DB issue not in JSONL:\n 1. Check deletions manifest → if found, delete from DB\n 2. Fallback: check git history → if found, delete + backfill manifest\n 3. Neither → keep (local unpushed work)\n```\n\n### Conflict Resolution\nSimultaneous deletions from multiple clones are handled naturally:\n- Append-only design means both clones append their deletion records\n- On merge, file contains duplicate entries (same ID, different timestamps)\n- `LoadDeletions` deduplicates by ID (keeps any/first entry)\n- Result: deletion propagates correctly, duplicates are harmless\n\n### Pruning Policy\n- Default retention: 7 days (configurable via `deletions.retention_days`)\n- Auto-compact during `bd sync` is **opt-in** (disabled by default)\n- Hard cap: `deletions.max_entries` (default 50000)\n- Git fallback handles pruned entries (self-healing)\n\n### Self-Healing\nWhen git fallback catches a resurrection (pruned entry), it backfills the manifest. One-time git scan cost, then fast again.\n\n### Size Estimates\n- ~80 bytes/entry\n- 7-day retention with 100 deletions/day = ~56KB\n- Git compressed: ~10KB\n\n## Benefits\n- ✅ Deletions propagate across clones\n- ✅ O(1) lookup (no git scan in normal case)\n- ✅ Works in shallow clones\n- ✅ Survives history rewrite\n- ✅ Audit trail (who deleted what when)\n- ✅ Self-healing via git fallback\n- ✅ Bounded size via time-based pruning\n\n## References\n- Investigation session: 2025-11-25\n- Related: bd-2q6d (stale database warnings)","status":"closed","priority":0,"issue_type":"epic","created_at":"2025-11-25T09:56:01.98027-08:00","updated_at":"2025-11-25T16:36:27.965168-08:00","closed_at":"2025-11-25T16:36:27.965168-08:00","dependencies":[{"issue_id":"bd-imj","depends_on_id":"bd-qsm","type":"blocks","created_at":"2025-11-25T09:57:42.821911-08:00","created_by":"daemon"},{"issue_id":"bd-imj","depends_on_id":"bd-x2i","type":"blocks","created_at":"2025-11-25T09:57:42.851712-08:00","created_by":"daemon"},{"issue_id":"bd-imj","depends_on_id":"bd-44e","type":"blocks","created_at":"2025-11-25T09:57:42.88154-08:00","created_by":"daemon"},{"issue_id":"bd-imj","depends_on_id":"bd-bhd","type":"blocks","created_at":"2025-11-25T14:56:23.675787-08:00","created_by":"daemon"},{"issue_id":"bd-imj","depends_on_id":"bd-bgs","type":"blocks","created_at":"2025-11-25T14:56:23.744648-08:00","created_by":"daemon"},{"issue_id":"bd-imj","depends_on_id":"bd-f0n","type":"blocks","created_at":"2025-11-25T14:56:23.80649-08:00","created_by":"daemon"},{"issue_id":"bd-imj","depends_on_id":"bd-v29","type":"blocks","created_at":"2025-11-25T14:56:23.864569-08:00","created_by":"daemon"},{"issue_id":"bd-imj","depends_on_id":"bd-mdw","type":"blocks","created_at":"2025-11-25T14:56:48.592492-08:00","created_by":"daemon"},{"issue_id":"bd-imj","depends_on_id":"bd-03r","type":"blocks","created_at":"2025-11-25T14:56:54.295851-08:00","created_by":"daemon"}]} @@ -427,7 +426,7 @@ {"id":"bd-r9iq","title":"purgeDeletedIssues still writes to deletions.jsonl during git-history-backfill","description":"In purgeDeletedIssues (lines 929-939), when a deletion is recovered from git history, we:\n1. Append to deletions.jsonl (backfill)\n2. Create a tombstone in DB\n\nWith the tombstone approach, writing to deletions.jsonl is redundant since the tombstone will be exported to JSONL. Consider removing the deletions.jsonl backfill once we're confident in the tombstone approach.\n\nThis is related to the Phase 2 migration (bd-vw8) - we should stop writing to deletions.jsonl entirely.","status":"open","priority":2,"issue_type":"task","created_at":"2025-12-07T01:40:42.581876-08:00","updated_at":"2025-12-07T01:40:42.581876-08:00","dependencies":[{"issue_id":"bd-r9iq","depends_on_id":"bd-dve","type":"blocks","created_at":"2025-12-07T01:41:28.285457-08:00","created_by":"daemon"}]} {"id":"bd-s0z","title":"Consider extracting error handling helpers","description":"Evaluate creating FatalError() and WarnError() helpers as suggested in ERROR_HANDLING.md to reduce boilerplate and enforce consistency. Prototype in a few files first to validate the approach.","status":"closed","priority":4,"issue_type":"task","created_at":"2025-11-24T00:28:57.248959-08:00","updated_at":"2025-11-30T10:50:04.275143-08:00","closed_at":"2025-11-28T23:28:00.886536-08:00"} {"id":"bd-s2t","title":"wish: a 'continue' or similar cmd/flag which means alter last issue","description":"so many time I create an issue and then have another thought: 'oh, before I did X and it crashed there was ZZZ happening' or 'actually that is P4 not P2'. It would be nice if when `bd {cmd}` is used without a {title} or {id} it just adds or updates the most recently touched issue.","status":"open","priority":3,"issue_type":"task","created_at":"2025-12-08T06:46:37.529160416-07:00","updated_at":"2025-12-08T06:46:37.529160416-07:00"} -{"id":"bd-s3v","title":"Add tombstone doctor checks","description":"Add bd doctor checks for: unmigrated deletions.jsonl, stale deletions.jsonl in Phase 3, tombstones expiring soon. Per design bd-dli.","status":"open","priority":2,"issue_type":"task","created_at":"2025-12-05T15:14:46.733558-08:00","updated_at":"2025-12-05T15:14:46.733558-08:00","dependencies":[{"issue_id":"bd-s3v","depends_on_id":"bd-8f9","type":"blocks","created_at":"2025-12-05T15:14:59.006029-08:00","created_by":"daemon"}]} +{"id":"bd-s3v","title":"Add tombstone doctor checks","description":"Add bd doctor checks for: unmigrated deletions.jsonl, stale deletions.jsonl in Phase 3, tombstones expiring soon. Per design bd-dli.","status":"closed","priority":2,"issue_type":"task","created_at":"2025-12-05T15:14:46.733558-08:00","updated_at":"2025-12-13T10:15:38.981577-08:00","closed_at":"2025-12-13T10:15:38.981577-08:00","dependencies":[{"issue_id":"bd-s3v","depends_on_id":"bd-8f9","type":"blocks","created_at":"2025-12-05T15:14:59.006029-08:00","created_by":"daemon"}]} {"id":"bd-saa","title":"Add index on deleted_at for TTL queries","description":"Per bd-2m7 design: 'Index for efficient tombstone filtering' was recommended. The current implementation does NOT add an index on deleted_at.\n\nFor TTL cleanup queries like 'SELECT * FROM issues WHERE deleted_at \u003c datetime(now, -30 days)', this will require a full table scan.\n\nAdd to migration or new migration:\nCREATE INDEX IF NOT EXISTS idx_issues_deleted_at ON issues(deleted_at) WHERE deleted_at IS NOT NULL","status":"closed","priority":2,"issue_type":"task","created_at":"2025-12-05T15:35:33.842664-08:00","updated_at":"2025-12-05T15:48:07.584268-08:00","closed_at":"2025-12-05T15:48:07.584268-08:00","dependencies":[{"issue_id":"bd-saa","depends_on_id":"bd-vw8","type":"parent-child","created_at":"2025-12-05T15:35:47.222711-08:00","created_by":"daemon"}]} {"id":"bd-t3b","title":"Add test coverage for internal/config package","description":"","status":"closed","priority":2,"issue_type":"task","created_at":"2025-11-20T21:21:22.91657-05:00","updated_at":"2025-11-30T10:50:04.275382-08:00","closed_at":"2025-11-28T21:54:15.009889-08:00","dependencies":[{"issue_id":"bd-t3b","depends_on_id":"bd-ge7","type":"blocks","created_at":"2025-11-20T21:21:31.201036-05:00","created_by":"daemon"}]} {"id":"bd-t5m","title":"CRITICAL: git-history-backfill purges entire database when JSONL reset","description":"When a clone gets reset (git reset --hard origin/main), the git-history-backfill logic incorrectly adds ALL issues to the deletions manifest, then sync purges the entire database.\\n\\nFix adds safety guard: never delete more than 50% of issues via git-history-backfill. If threshold exceeded, abort with warning message.","status":"closed","priority":0,"issue_type":"bug","created_at":"2025-11-30T21:24:47.397394-08:00","updated_at":"2025-11-30T21:24:52.710971-08:00","closed_at":"2025-11-30T21:24:52.710971-08:00"} diff --git a/cmd/bd/doctor.go b/cmd/bd/doctor.go index f5b0cd87..49e798d3 100644 --- a/cmd/bd/doctor.go +++ b/cmd/bd/doctor.go @@ -766,12 +766,17 @@ func runDiagnostics(path string) doctorResult { result.Checks = append(result.Checks, syncBranchHealthCheck) // Don't fail overall check for sync branch health, just warn - // Check 18: Deletions manifest (prevents zombie resurrection) + // Check 18: Deletions manifest (legacy, now replaced by tombstones) deletionsCheck := checkDeletionsManifest(path) result.Checks = append(result.Checks, deletionsCheck) // Don't fail overall check for missing deletions manifest, just warn - // Check 19: Untracked .beads/*.jsonl files (bd-pbj) + // Check 19: Tombstones health (bd-s3v) + tombstonesCheck := checkTombstones(path) + result.Checks = append(result.Checks, tombstonesCheck) + // Don't fail overall check for tombstone issues, just warn + + // Check 20: Untracked .beads/*.jsonl files (bd-pbj) untrackedCheck := checkUntrackedBeadsFiles(path) result.Checks = append(result.Checks, untrackedCheck) // Don't fail overall check for untracked files, just warn @@ -2557,7 +2562,7 @@ func checkDeletionsManifest(path string) doctorCheck { return doctorCheck{ Name: "Deletions Manifest", Status: statusOK, - Message: "Present (0 entries)", + Message: "Empty (no legacy deletions)", } } file, err := os.Open(deletionsPath) // #nosec G304 - controlled path @@ -2570,17 +2575,36 @@ func checkDeletionsManifest(path string) doctorCheck { count++ } } + // bd-s3v: Suggest migration to inline tombstones + if count > 0 { + return doctorCheck{ + Name: "Deletions Manifest", + Status: statusWarning, + Message: fmt.Sprintf("Legacy format (%d entries)", count), + Detail: "deletions.jsonl is deprecated in favor of inline tombstones", + Fix: "Run 'bd migrate-tombstones' to convert to inline tombstones", + } + } return doctorCheck{ Name: "Deletions Manifest", Status: statusOK, - Message: fmt.Sprintf("Present (%d entries)", count), + Message: "Empty (no legacy deletions)", } } } - // deletions.jsonl doesn't exist or is empty - // Check if there's git history that might have deletions - // bd-6xd: Check canonical issues.jsonl first, then legacy beads.jsonl + // bd-s3v: deletions.jsonl doesn't exist - this is the expected state with tombstones + // Check for .migrated file to confirm migration happened + migratedPath := filepath.Join(beadsDir, "deletions.jsonl.migrated") + if _, err := os.Stat(migratedPath); err == nil { + return doctorCheck{ + Name: "Deletions Manifest", + Status: statusOK, + Message: "Migrated to tombstones", + } + } + + // No deletions.jsonl and no .migrated file - check if JSONL exists jsonlPath := filepath.Join(beadsDir, "issues.jsonl") if _, err := os.Stat(jsonlPath); os.IsNotExist(err) { jsonlPath = filepath.Join(beadsDir, "beads.jsonl") @@ -2593,26 +2617,110 @@ func checkDeletionsManifest(path string) doctorCheck { } } - // Check if JSONL has any git history - relPath, _ := filepath.Rel(path, jsonlPath) - cmd := exec.Command("git", "log", "--oneline", "-1", "--", relPath) // #nosec G204 - args are controlled - cmd.Dir = path - if output, err := cmd.Output(); err != nil || len(output) == 0 { - // No git history for JSONL + // JSONL exists but no deletions tracking - this is fine for new repos using tombstones + return doctorCheck{ + Name: "Deletions Manifest", + Status: statusOK, + Message: "Using inline tombstones", + } +} + +// checkTombstones checks the health of tombstone records (bd-s3v) +// Reports: total tombstones, expiring soon (within 7 days), already expired +func checkTombstones(path string) doctorCheck { + beadsDir := filepath.Join(path, ".beads") + dbPath := filepath.Join(beadsDir, beads.CanonicalDatabaseName) + + // Skip if database doesn't exist + if _, err := os.Stat(dbPath); os.IsNotExist(err) { return doctorCheck{ - Name: "Deletions Manifest", + Name: "Tombstones", Status: statusOK, - Message: "Not yet created (no deletions recorded)", + Message: "N/A (no database)", + } + } + + db, err := sql.Open("sqlite3", dbPath) + if err != nil { + return doctorCheck{ + Name: "Tombstones", + Status: statusWarning, + Message: "Unable to open database", + Detail: err.Error(), + } + } + defer db.Close() + + // Query tombstone statistics + var totalTombstones int + err = db.QueryRow("SELECT COUNT(*) FROM issues WHERE status = 'tombstone'").Scan(&totalTombstones) + if err != nil { + // Might be old schema without tombstone support + return doctorCheck{ + Name: "Tombstones", + Status: statusOK, + Message: "N/A (schema may not support tombstones)", + } + } + + if totalTombstones == 0 { + return doctorCheck{ + Name: "Tombstones", + Status: statusOK, + Message: "None (no deleted issues)", + } + } + + // Check for tombstones expiring within 7 days + // Default TTL is 30 days, so expiring soon means deleted_at older than 23 days ago + expiringThreshold := time.Now().Add(-23 * 24 * time.Hour).Format(time.RFC3339) + expiredThreshold := time.Now().Add(-30 * 24 * time.Hour).Format(time.RFC3339) + + var expiringSoon, alreadyExpired int + err = db.QueryRow(` + SELECT COUNT(*) FROM issues + WHERE status = 'tombstone' + AND deleted_at IS NOT NULL + AND deleted_at < ? + AND deleted_at >= ? + `, expiringThreshold, expiredThreshold).Scan(&expiringSoon) + if err != nil { + expiringSoon = 0 + } + + err = db.QueryRow(` + SELECT COUNT(*) FROM issues + WHERE status = 'tombstone' + AND deleted_at IS NOT NULL + AND deleted_at < ? + `, expiredThreshold).Scan(&alreadyExpired) + if err != nil { + alreadyExpired = 0 + } + + // Build status message + if alreadyExpired > 0 { + return doctorCheck{ + Name: "Tombstones", + Status: statusWarning, + Message: fmt.Sprintf("%d total, %d expired", totalTombstones, alreadyExpired), + Detail: "Expired tombstones will be removed on next compact", + Fix: "Run 'bd compact' to prune expired tombstones", + } + } + + if expiringSoon > 0 { + return doctorCheck{ + Name: "Tombstones", + Status: statusOK, + Message: fmt.Sprintf("%d total, %d expiring within 7 days", totalTombstones, expiringSoon), } } - // There's git history but no deletions manifest - recommend hydration return doctorCheck{ - Name: "Deletions Manifest", - Status: statusWarning, - Message: "Missing or empty (may have pre-v0.25.0 deletions)", - Detail: "Deleted issues from before v0.25.0 are not tracked and may resurrect on sync", - Fix: "Run 'bd doctor --fix' to hydrate deletions manifest from git history", + Name: "Tombstones", + Status: statusOK, + Message: fmt.Sprintf("%d total", totalTombstones), } }