johno/beads
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement incremental JSONL export with dirty issue tracking

Browse Source 
Optimize auto-flush by tracking which issues have changed instead of
exporting the entire database on every flush. For large projects with
1000+ issues, this provides significant performance improvements.

Changes:
- Add dirty_issues table to schema with issue_id and marked_at columns
- Implement dirty tracking functions in new dirty.go file:
  * MarkIssueDirty() - Mark single issue as needing export
  * MarkIssuesDirty() - Batch mark multiple issues efficiently
  * GetDirtyIssues() - Query which issues need export
  * ClearDirtyIssues() - Clear tracking after successful export
  * GetDirtyIssueCount() - Monitor dirty issue count
- Update all CRUD operations to mark affected issues as dirty:
  * CreateIssue, UpdateIssue, DeleteIssue
  * AddDependency, RemoveDependency (marks both issues)
  * AddLabel, RemoveLabel, AddEvent
- Modify export to support incremental mode:
  * Add --incremental flag to export only dirty issues
  * Used by auto-flush for performance
  * Full export still available without flag
- Add Storage interface methods for dirty tracking

Performance impact: With incremental export, large databases only write
changed issues instead of regenerating entire JSONL file on every
auto-flush.

Closes bd-39

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Steve Yegge
2025-10-14 00:17:23 -07:00
parent 25644d9717
commit bafb2801c5
11 changed files with 372 additions and 99 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
.beads/bd.jsonl
View File

@@ -30,7 +30,7 @@
{"id":"bd-36","title":"Handle missing JSONL directory in findJSONLPath","description":"findJSONLPath() assumes the database directory exists. If someone runs bd init to create a new database but the .beads directory doesn't exist yet, the glob operations might fail silently. Add os.MkdirAll(dbDir, 0755) to ensure directory exists before globbing. Located in cmd/bd/main.go:188-201.","status":"closed","priority":2,"issue_type":"bug","created_at":"2025-10-13T22:34:35.944346-07:00","updated_at":"2025-10-13T23:26:35.811504-07:00","closed_at":"2025-10-13T22:50:53.269614-07:00"}
{"id":"bd-37","title":"Refactor duplicate flush logic in PersistentPostRun","description":"PersistentPostRun contains a complete copy of the flush logic instead of calling flushToJSONL(). This violates DRY principle and makes maintenance harder. Refactor to use flushToJSONL() with a force parameter to bypass isDirty check, or extract shared logic into a helper function. Located in cmd/bd/main.go:104-138.","status":"open","priority":3,"issue_type":"task","created_at":"2025-10-13T22:34:43.429201-07:00","updated_at":"2025-10-13T23:26:35.811582-07:00"}
{"id":"bd-38","title":"Add visibility for auto-flush failures","description":"flushToJSONL() writes warnings to stderr when flush fails, but calling code has no way to know if flush succeeded or failed. This means a command could return success even though JSONL is now out of sync. Consider maintaining a 'last flush status' variable or counter for failed flushes, and warn user after multiple consecutive failures (e.g., 3+). Located in cmd/bd/main.go:227-307.","status":"closed","priority":2,"issue_type":"feature","created_at":"2025-10-13T22:34:52.440117-07:00","updated_at":"2025-10-13T23:26:35.811675-07:00","closed_at":"2025-10-13T23:22:47.805211-07:00"}
{"id":"bd-39","title":"Optimize auto-flush to use incremental updates","description":"Every flush exports ALL issues and ALL dependencies, even if only one issue changed. For large projects (1000+ issues), this could be expensive. Current approach guarantees consistency, which is fine for MVP, but future optimization could track which issues changed and use incremental updates. Located in cmd/bd/main.go:255-276.","status":"open","priority":3,"issue_type":"feature","created_at":"2025-10-13T22:34:59.26425-07:00","updated_at":"2025-10-13T23:26:35.811755-07:00"}
{"id":"bd-39","title":"Optimize auto-flush to use incremental updates","description":"Every flush exports ALL issues and ALL dependencies, even if only one issue changed. For large projects (1000+ issues), this could be expensive. Current approach guarantees consistency, which is fine for MVP, but future optimization could track which issues changed and use incremental updates. Located in cmd/bd/main.go:255-276.","status":"closed","priority":3,"issue_type":"feature","created_at":"2025-10-13T22:34:59.26425-07:00","updated_at":"2025-10-14T00:08:51.834812-07:00","closed_at":"2025-10-14T00:08:51.834812-07:00"}
{"id":"bd-4","title":"Add demo GIF/video showing bd quickstart in action","description":"Record asciinema or create animated GIF showing the full workflow","status":"open","priority":2,"issue_type":"feature","created_at":"2025-10-12T10:50:49.500051-07:00","updated_at":"2025-10-13T23:26:35.811831-07:00","dependencies":[{"issue_id":"bd-4","depends_on_id":"bd-8","type":"parent-child","created_at":"2025-10-12T10:51:08.399915-07:00","created_by":"stevey"}]}
{"id":"bd-40","title":"Make auto-flush debounce duration configurable","description":"flushDebounce is hardcoded to 5 seconds. Make it configurable via environment variable BEADS_FLUSH_DEBOUNCE (e.g., '500ms', '10s'). Current 5-second value is reasonable for interactive use, but CI/automated scenarios might want faster flush. Add getDebounceDuration() helper function. Located in cmd/bd/main.go:31.","status":"open","priority":3,"issue_type":"feature","created_at":"2025-10-13T22:35:06.126282-07:00","updated_at":"2025-10-13T23:26:35.81192-07:00"}
{"id":"bd-41","title":"Add godoc comments for auto-flush functions","description":"Add comprehensive godoc comments for findJSONLPath(), markDirtyAndScheduleFlush(), and flushToJSONL() explaining behavior, concurrency considerations, and error handling. Include notes about debouncing behavior (timer resets on each write, flush occurs 5s after LAST operation) and flush-on-exit guarantees. Located in cmd/bd/main.go:188-307.","status":"open","priority":4,"issue_type":"chore","created_at":"2025-10-13T22:35:13.518442-07:00","updated_at":"2025-10-13T23:26:35.811999-07:00"}
@@ -39,6 +39,9 @@
{"id":"bd-44","title":"Regular auto-ID issue","description":"","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-13T23:16:29.970089-07:00","updated_at":"2025-10-13T23:26:35.812252-07:00","closed_at":"2025-10-13T23:16:45.231439-07:00"}
{"id":"bd-45","title":"Test flush tracking","description":"","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-13T23:22:18.472476-07:00","updated_at":"2025-10-13T23:26:35.812337-07:00","closed_at":"2025-10-13T23:22:31.397095-07:00"}
{"id":"bd-46","title":"Test export cancels timer","description":"","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-13T23:26:20.523923-07:00","updated_at":"2025-10-13T23:26:35.813165-07:00","closed_at":"2025-10-13T23:26:35.813165-07:00"}
{"id":"bd-47","title":"Test incremental export","description":"Testing bd-39 implementation","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-14T00:06:24.42044-07:00","updated_at":"2025-10-14T00:06:24.42044-07:00"}
{"id":"bd-48","title":"Test incremental 2","description":"","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-14T00:07:14.157987-07:00","updated_at":"2025-10-14T00:07:14.157987-07:00"}
{"id":"bd-49","title":"Final test","description":"Testing with new binary","status":"in_progress","priority":1,"issue_type":"task","created_at":"2025-10-14T00:07:46.650341-07:00","updated_at":"2025-10-14T00:08:23.657651-07:00"}
{"id":"bd-5","title":"Implement MCP server for Claude Desktop","description":"Complete the claude-desktop-mcp example with working TypeScript implementation","status":"closed","priority":1,"issue_type":"feature","created_at":"2025-10-12T10:50:50.942964-07:00","updated_at":"2025-10-13T23:26:35.8125-07:00","closed_at":"2025-10-13T23:20:41.816853-07:00","dependencies":[{"issue_id":"bd-5","depends_on_id":"bd-8","type":"parent-child","created_at":"2025-10-12T10:51:08.404381-07:00","created_by":"stevey"}]}
{"id":"bd-6","title":"Add migration scripts for GitHub Issues","description":"Create scripts to import from GitHub Issues API or exported JSON","status":"open","priority":2,"issue_type":"feature","created_at":"2025-10-12T10:50:52.140018-07:00","updated_at":"2025-10-13T23:26:35.81259-07:00","dependencies":[{"issue_id":"bd-6","depends_on_id":"bd-8","type":"parent-child","created_at":"2025-10-12T10:51:08.40857-07:00","created_by":"stevey"}]}
{"id":"bd-7","title":"Add performance benchmarks document","description":"Document actual performance metrics with hyperfine tests","status":"open","priority":3,"issue_type":"task","created_at":"2025-10-12T10:50:53.294516-07:00","updated_at":"2025-10-13T23:26:35.812667-07:00","dependencies":[{"issue_id":"bd-7","depends_on_id":"bd-8","type":"parent-child","created_at":"2025-10-12T10:51:08.412835-07:00","created_by":"stevey"}]}