Commit Graph

24 Commits

Author SHA1 Message Date
Steve Yegge
ccdacf087b fix: Auto-export to JSONL now works correctly [fixes #23]
Fixed bug where PersistentPostRun was clearing isDirty flag before
calling flushToJSONL(), causing the flush to abort immediately.

The fix ensures flushToJSONL() handles the isDirty flag itself,
allowing the JSONL export to complete successfully.

Also added Arch Linux AUR installation instructions to README.

Changes:
- cmd/bd/main.go: Fixed PersistentPostRun flush logic
- README.md: Added Arch Linux (AUR) installation section
- .beads/bd.jsonl: Auto-exported issue bd-169 (init -q flag bug)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 12:35:48 -07:00
Steve Yegge
2a093cad5c chore: Update JSONL export after bd-85 fix 2025-10-14 03:12:16 -07:00
Steve Yegge
88b4d710eb feat: Create bd tracking issues for 9 GitHub issues
Created internal tracking issues for all open GitHub Issues:
- bd-85 (gh-1): Fix dep tree graph display issues [P1]
- bd-86 (gh-2): Evaluate Turso backend RFC [P3]
- bd-87 (gh-3): Debug zsh killed error [P1]
- bd-88 (gh-4): System-wide/multi-repo usage [P3]
- bd-89 (gh-6): Fix parallel creation race condition [P0]
- bd-90 (gh-7): AUR package tracking [P4]
- bd-91 (gh-9): Markdown file input support [P2]
- bd-92 (gh-11): Docker/hosted instance support [P2]
- bd-93 (gh-18): Add --deps flag to create [P2]

All issues use the new external_ref field to link to their GitHub
counterparts (gh-N). This establishes proper bidirectional tracking
between our internal beads workflow and public GitHub Issues.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 02:48:00 -07:00
Steve Yegge
6456fe6c02 chore: Update JSONL export after external_ref implementation
Auto-sync export after implementing the external_ref field feature.
All existing issues remain unchanged (field not populated yet).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 02:43:52 -07:00
Steve Yegge
bafb2801c5 Implement incremental JSONL export with dirty issue tracking
Optimize auto-flush by tracking which issues have changed instead of
exporting the entire database on every flush. For large projects with
1000+ issues, this provides significant performance improvements.

Changes:
- Add dirty_issues table to schema with issue_id and marked_at columns
- Implement dirty tracking functions in new dirty.go file:
  * MarkIssueDirty() - Mark single issue as needing export
  * MarkIssuesDirty() - Batch mark multiple issues efficiently
  * GetDirtyIssues() - Query which issues need export
  * ClearDirtyIssues() - Clear tracking after successful export
  * GetDirtyIssueCount() - Monitor dirty issue count
- Update all CRUD operations to mark affected issues as dirty:
  * CreateIssue, UpdateIssue, DeleteIssue
  * AddDependency, RemoveDependency (marks both issues)
  * AddLabel, RemoveLabel, AddEvent
- Modify export to support incremental mode:
  * Add --incremental flag to export only dirty issues
  * Used by auto-flush for performance
  * Full export still available without flag
- Add Storage interface methods for dirty tracking

Performance impact: With incremental export, large databases only write
changed issues instead of regenerating entire JSONL file on every
auto-flush.

Closes bd-39

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 00:17:23 -07:00
Steve Yegge
25644d9717 Cache compiled regexes in ID replacement for 1.9x performance boost
Implements bd-27: Cache compiled regexes in replaceIDReferences for performance

Problem:
replaceIDReferences() was compiling regex patterns on every call. With 100
issues and 10 ID mappings, that resulted in 4,000 regex compilations (100
issues × 4 text fields × 10 ID mappings).

Solution:
- Added buildReplacementCache() to pre-compile all regexes once
- Added replaceIDReferencesWithCache() to reuse compiled regexes
- Updated updateReferences() to build cache once and reuse for all issues
- Kept replaceIDReferences() for backward compatibility (calls cached version)

Performance Results (from benchmarks):
Single text:
- 1.33x faster (26,162 ns → 19,641 ns)
- 68% less memory (25,769 B → 8,241 B)
- 80% fewer allocations (278 → 55)

Real-world (400 texts, 10 mappings):
- 1.89x faster (5.1ms → 2.7ms)
- 90% less memory (7.7 MB → 0.8 MB)
- 86% fewer allocations (104,112 → 14,801)

Tests:
- All existing tests pass
- Added 3 benchmark tests demonstrating improvements

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 23:50:48 -07:00
Steve Yegge
33412871eb Add comprehensive test coverage for auto-flush feature
Implements bd-42: Add test coverage for auto-flush feature

Created cmd/bd/main_test.go with 11 comprehensive test functions:
- TestAutoFlushDirtyMarking: Verifies markDirtyAndScheduleFlush() marks DB as dirty
- TestAutoFlushDisabled: Tests --no-auto-flush flag disables feature
- TestAutoFlushDebounce: Tests rapid operations result in single flush
- TestAutoFlushClearState: Tests clearAutoFlushState() resets state
- TestAutoFlushOnExit: Tests flush happens on program exit
- TestAutoFlushConcurrency: Tests concurrent operations don't cause races
- TestAutoFlushStoreInactive: Tests flush skips when store is inactive
- TestAutoFlushJSONLContent: Tests flushed JSONL has correct content
- TestAutoFlushErrorHandling: Tests error scenarios (permissions, etc.)
- TestAutoImportIfNewer: Tests auto-import when JSONL is newer than DB
- TestAutoImportDisabled: Tests --no-auto-import flag disables auto-import

Coverage results:
- markDirtyAndScheduleFlush: 100%
- clearAutoFlushState: 100%
- flushToJSONL: 67.6%
- autoImportIfNewer: 66.1% (up from 0%)

All tests pass. Auto-flush feature is now thoroughly tested.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 23:38:24 -07:00
Steve Yegge
a8a90e074e Add ID space partitioning and improve auto-flush reliability
Three improvements to beads:

1. ID space partitioning (closes bd-24)
   - Add --id flag to 'bd create' for explicit ID assignment
   - Validates format: prefix-number (e.g., worker1-100)
   - Enables parallel agents to partition ID space and avoid conflicts
   - Storage layer already supported this, just wired up CLI

2. Auto-flush failure tracking (closes bd-38)
   - Track consecutive flush failures with counter and last error
   - Show prominent red warning after 3+ consecutive failures
   - Reset counter on successful flush
   - Users get clear guidance to run manual export if needed

3. Manual export cancels auto-flush timer
   - Add clearAutoFlushState() helper function
   - bd export now cancels pending auto-flush and clears dirty flag
   - Prevents redundant exports when user manually exports
   - Also resets failure counter on successful manual export

Documentation updated in README.md and CLAUDE.md with --id flag examples.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 23:31:51 -07:00
Steve Yegge
252cf9a192 Update issue tracker - close bd-25 as won't-fix
Downgraded bd-25 to P4 and closed as won't-fix for 1.0:
- Transaction support is premature optimization
- SQLite already provides ACID guarantees per-operation
- Collision resolution works reliably without multi-operation transactions
- Would add significant complexity for theoretical benefit

Will revisit if actual issues arise in production use.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 22:59:24 -07:00
Steve Yegge
12a4c384af Fix: Ensure JSONL directory exists in findJSONLPath
Added os.MkdirAll(dbDir, 0755) to ensure the .beads directory exists
before attempting to glob for JSONL files. This fixes a bug where
findJSONLPath() would fail silently if the directory doesn't exist yet,
which can happen during new database initialization.

The fix:
- Creates the directory with 0755 permissions if it doesn't exist
- Handles errors gracefully by returning the default path
- Subsequent write operations will still fail with clear errors if
  directory creation fails

Closes bd-36

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 22:51:04 -07:00
Steve Yegge
584cd1ebfc Implement auto-import to complete automatic git sync workflow
Adds auto-import feature to complement bd-35's auto-export, completing
the automatic sync workflow for git collaboration.

**Implementation:**
- Auto-import checks if JSONL is newer than DB on command startup
- Silently imports JSONL when modification time is newer
- Skips import command itself to avoid recursion
- Can be disabled with --no-auto-import flag

**Documentation updates:**
- Updated README.md git workflow section
- Updated CLAUDE.md workflow and pro tips
- Updated bd quickstart with auto-sync section
- Updated git hooks README to clarify they're now optional

**Testing:**
- Tested auto-import by touching JSONL and running commands
- Tested auto-export with create/close operations
- Complete workflow verified working

Closes bd-33

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 22:48:30 -07:00
Steve Yegge
026940c8ae Track code review findings as issues (bd-36 through bd-42)
Added issues tracking code quality findings from the recent auto-flush implementation:
- bd-36: Handle missing JSONL directory in findJSONLPath
- bd-37: Refactor duplicate flush logic in PersistentPostRun
- bd-38: Add visibility for auto-flush failures
- bd-39: Optimize auto-flush to use incremental updates
- bd-40: Make auto-flush debounce duration configurable
- bd-41: Add godoc comments for auto-flush functions
- bd-42: Add test coverage for auto-flush feature

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 22:37:52 -07:00
Steve Yegge
37f3a8da87 Track code review findings as issues (bd-36 through bd-42) 2025-10-13 22:35:29 -07:00
Steve Yegge
97d78d264f Fix critical race conditions in auto-flush feature
Fixed three critical issues identified in code review:

1. Race condition with store access: Added storeMutex and storeActive
   flag to prevent background flush goroutine from accessing closed
   store. Background timer now safely checks if store is active before
   attempting flush operations.

2. Missing auto-flush in import: Added markDirtyAndScheduleFlush()
   call after import completes, ensuring imported issues sync to JSONL.

3. Timer cleanup: Explicitly set flushTimer to nil after Stop() to
   prevent resource leaks.

Testing confirmed all fixes working:
- Debounced flush triggers after 5 seconds of inactivity
- Immediate flush on process exit works correctly
- Import operations now trigger auto-flush
- No race conditions detected

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 22:34:12 -07:00
Steve Yegge
aaeff4d9bc Close bd-1: Export/import commands fully implemented
Feature is complete with JSONL format, collision detection, and auto-resolution.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 21:58:43 -07:00
Steve Yegge
183ded4096 Add collision resolution with automatic ID remapping
Implements --resolve-collisions flag for import command to safely handle ID
collisions during branch merges. When enabled, colliding issues are remapped
to new IDs and all text references and dependencies are automatically updated.

Also adds comprehensive tests, branch-merge example, and documentation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 17:13:09 -07:00
Steve Yegge
42e3bb315d Implement reference scoring algorithm (bd-13)
Add reference scoring to prioritize which colliding issues should be
renumbered during collision resolution. Issues with fewer references
are renumbered first to minimize total update work.

Changes to collision.go:
- Add ReferenceScore field to CollisionDetail
- scoreCollisions() calculates scores and sorts collisions ascending
- countReferences() counts text mentions + dependency references
- Uses word-boundary regex (\b) to match exact IDs (bd-10 not bd-100)

New tests in collision_test.go:
- TestCountReferences: validates reference counting logic
- TestScoreCollisions: verifies scoring and sorting behavior
- TestCountReferencesWordBoundary: ensures exact ID matching

Reference score = text mentions (desc/design/notes/criteria) + deps
Sort order: fewest references first (minimizes renumbering impact)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 16:27:05 -07:00
Steve Yegge
ee14c811c4 Add bd-24: Support ID space partitioning for parallel worker agents
Enable external orchestrators to control issue ID assignment via --id flag
on bd create and optional next_id config. Keeps beads simple while allowing
orchestrators to implement ID partitioning strategies to minimize merge
conflicts. Complementary to bd-9 collision resolution work.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 16:20:21 -07:00
Steve Yegge
3440899850 Fix code review issues bd-19 through bd-23
- bd-19: Fix import zero-value field handling using JSON presence detection
- bd-20: Add --strict flag for dependency import failures
- bd-21: Simplify getNextID SQL query (reduce params from 4 to 2)
- bd-22: Add validation/warning for malformed issue IDs
- bd-23: Optimize export dependency queries (fix N+1 problem)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 15:30:11 -07:00
Steve Yegge
f048602d9f Add code review issues (bd-19 to bd-23) 2025-10-12 15:13:37 -07:00
Steve Yegge
8d3bbbcc52 Implement bd-10: Export/import dependencies in JSONL
Added dependency support to export/import workflow:

Changes:
- Added GetDependencyRecords() to Storage interface to get raw dependency records
- Extended Issue struct with Dependencies field (omitempty for backward compat)
- Modified export.go to populate dependencies for each issue
- Modified import.go to process dependencies in second pass after all issues exist
- All tests pass

Benefits:
- JSONL is now self-contained with full dependency information
- Enables proper collision resolution in future (bd-12+)
- Idempotent imports: existing dependencies are not duplicated
- Forward references handled: dependencies created after all issues exist

Example output:
{
  "id": "bd-10",
  "title": "...",
  "dependencies": [{
    "issue_id": "bd-10",
    "depends_on_id": "bd-9",
    "type": "parent-child",
    "created_at": "...",
    "created_by": "stevey"
  }]
}

Closes bd-10

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 15:08:05 -07:00
Steve Yegge
216f640ab4 Fix getNextID bug and design collision resolution (bd-9)
Critical bug fix: getNextID() was using alphabetical MAX instead of
numerical MAX, causing "bd-9" to be treated as max when "bd-10" existed.
This blocked all new issue creation after bd-10.

Fixed by using SQL CAST to extract and compare numeric portions of IDs.
This ensures bd-10 > bd-9 numerically, not alphabetically.

Also completed comprehensive design for bd-9 (collision resolution):
- Algorithm design with 7 phases (detection, scoring, remapping, etc.)
- Created 7 child issues (bd-10, bd-12-17) breaking down implementation
- Added design documents to .beads/ for future reference
- Updated issues JSONL with new issues and dependencies

Issues created:
- bd-10: Export dependencies in JSONL
- bd-12: Collision detection
- bd-13: Reference scoring algorithm
- bd-14: ID remapping with updates
- bd-15: CLI flags and reporting
- bd-16: Comprehensive tests
- bd-17: Documentation updates
- bd-18: Add design/notes fields to update command

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 14:41:58 -07:00
Steve Yegge
19cd7d1887 Prepare for public launch: comprehensive examples, docs, and tooling
This commit adds everything needed for a successful public launch:

**New Documentation**
- SECURITY.md: Security policy and best practices
- CLAUDE.md: Complete agent instructions for contributing to beads
- Enhanced README with pain points, FAQ, troubleshooting sections
- Added Taskwarrior to comparison table with detailed explanation

**Installation**
- install.sh: One-liner installation script with platform detection
- Auto-detects OS/arch, tries go install, falls back to building from source
- Updated README with prominent installation instructions

**Examples** (2,268+ lines of working code)
- examples/python-agent/: Full Python implementation of agent workflow
- examples/bash-agent/: Shell script agent with colorized output
- examples/git-hooks/: Pre-commit, post-merge, post-checkout hooks with installer
- examples/claude-desktop-mcp/: Documentation for future MCP server integration
- examples/README.md: Overview of all examples

**Dogfooding**
- Initialized bd in beads project itself (.beads/beads.db)
- Created issues for roadmap (MCP server, migrations, demos, 1.0 milestone)
- Exported to .beads/issues.jsonl for git versioning

**Visual Assets**
- Added screenshot showing agent using beads to README intro
- Placed in .github/images/ following GitHub conventions

This addresses all launch readiness items:
 Security policy
 Working agent examples (Python, Bash)
 Git hooks for automation
 FAQ addressing skeptics
 Troubleshooting common issues
 Easy installation
 Dogfooding our own tool
 Pain points that create urgency

Ready to ship! 🚀

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 11:25:29 -07:00
Steve Yegge
15afb5ad17 Implement JSONL export/import and shift to text-first architecture
This is a fundamental architectural shift from binary SQLite to JSONL as
the source of truth for git workflows.

## New Features

- `bd export --format=jsonl` - Export issues to JSON Lines format
- `bd import` - Import issues from JSONL (create new, update existing)
- `--skip-existing` flag for import to only create new issues

## Architecture Change

**Before:** Binary SQLite database committed to git
**After:** JSONL text files as source of truth, SQLite as ephemeral cache

Benefits:
- Git-friendly text format with clean diffs
- AI-resolvable merge conflicts (append-only is 95% conflict-free)
- Human-readable issue tracking in git
- No binary merge conflicts

## Documentation

- Updated README with JSONL-first workflow and git hooks
- Added TEXT_FORMATS.md analyzing JSONL vs CSV vs binary
- Updated GIT_WORKFLOW.md with historical context
- .gitignore now excludes *.db, includes .beads/*.jsonl

## Implementation Details

- Export sorts issues by ID for consistent diffs
- Import handles both creates and updates atomically
- Proper handling of pointer fields (EstimatedMinutes)
- All tests passing

## Breaking Changes

- Database files (*.db) should now be gitignored
- Use export/import workflow for git collaboration
- Git hooks recommended for automation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 01:17:50 -07:00