Implement incremental JSONL export with dirty issue tracking

Optimize auto-flush by tracking which issues have changed instead of
exporting the entire database on every flush. For large projects with
1000+ issues, this provides significant performance improvements.

Changes:
- Add dirty_issues table to schema with issue_id and marked_at columns
- Implement dirty tracking functions in new dirty.go file:
  * MarkIssueDirty() - Mark single issue as needing export
  * MarkIssuesDirty() - Batch mark multiple issues efficiently
  * GetDirtyIssues() - Query which issues need export
  * ClearDirtyIssues() - Clear tracking after successful export
  * GetDirtyIssueCount() - Monitor dirty issue count
- Update all CRUD operations to mark affected issues as dirty:
  * CreateIssue, UpdateIssue, DeleteIssue
  * AddDependency, RemoveDependency (marks both issues)
  * AddLabel, RemoveLabel, AddEvent
- Modify export to support incremental mode:
  * Add --incremental flag to export only dirty issues
  * Used by auto-flush for performance
  * Full export still available without flag
- Add Storage interface methods for dirty tracking

Performance impact: With incremental export, large databases only write
changed issues instead of regenerating entire JSONL file on every
auto-flush.

Closes bd-39

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Steve Yegge
2025-10-14 00:17:23 -07:00
parent 25644d9717
commit bafb2801c5
11 changed files with 372 additions and 99 deletions

View File

@@ -173,6 +173,16 @@ func (s *SQLiteStorage) CreateIssue(ctx context.Context, issue *types.Issue, act
return fmt.Errorf("failed to record event: %w", err)
}
// Mark issue as dirty for incremental export
_, err = tx.ExecContext(ctx, `
INSERT INTO dirty_issues (issue_id, marked_at)
VALUES (?, ?)
ON CONFLICT (issue_id) DO UPDATE SET marked_at = excluded.marked_at
`, issue.ID, time.Now())
if err != nil {
return fmt.Errorf("failed to mark issue dirty: %w", err)
}
return tx.Commit()
}
@@ -336,6 +346,16 @@ func (s *SQLiteStorage) UpdateIssue(ctx context.Context, id string, updates map[
return fmt.Errorf("failed to record event: %w", err)
}
// Mark issue as dirty for incremental export
_, err = tx.ExecContext(ctx, `
INSERT INTO dirty_issues (issue_id, marked_at)
VALUES (?, ?)
ON CONFLICT (issue_id) DO UPDATE SET marked_at = excluded.marked_at
`, id, time.Now())
if err != nil {
return fmt.Errorf("failed to mark issue dirty: %w", err)
}
return tx.Commit()
}
@@ -366,6 +386,16 @@ func (s *SQLiteStorage) CloseIssue(ctx context.Context, id string, reason string
return fmt.Errorf("failed to record event: %w", err)
}
// Mark issue as dirty for incremental export
_, err = tx.ExecContext(ctx, `
INSERT INTO dirty_issues (issue_id, marked_at)
VALUES (?, ?)
ON CONFLICT (issue_id) DO UPDATE SET marked_at = excluded.marked_at
`, id, time.Now())
if err != nil {
return fmt.Errorf("failed to mark issue dirty: %w", err)
}
return tx.Commit()
}