Commit Graph

18 Commits

Author SHA1 Message Date
Steve Yegge
a02729ea57 Complete bd-164: Fix timestamp-only export deduplication
- Add export_hashes table to track last exported content state
- Implement GetExportHash/SetExportHash in storage interface
- Update shouldSkipExport to use export_hashes instead of dirty_issues
- Call SetExportHash after successful export
- Clean up dirty_issues table (remove content_hash column)
- Simplify MarkIssueDirty functions (no longer compute hashes)
- Update markIssuesDirtyTx signature (remove store parameter)

Testing:
- Timestamp-only updates are skipped during export ✓
- Real content changes trigger export ✓
- export_hashes table populated correctly ✓

Fixes bd-159, bd-164
2025-10-26 20:35:37 -07:00
Steve Yegge
a898df6915 WIP: bd-164 timestamp-only export deduplication (~80% complete)
Implemented content hash-based deduplication to skip exports when only
timestamps changed. Core logic complete, needs export_hashes table wiring.

Completed:
- Added computeIssueContentHash() excluding timestamps
- Created shouldSkipExport() logic
- Updated export loop to skip timestamp-only changes
- Added hash.go with content hashing
- Extended Storage interface

Remaining:
- Complete export_hashes table migration
- Add SetExportHash/GetExportHash to interface
- Revert content_hash from dirty_issues approach
- Wire up hash persistence in export
- Testing

See bd-164 notes for details.

Amp-Thread-ID: https://ampcode.com/threads/T-d70657d1-4433-4f7e-b10a-3fccf8bf17fb
Co-authored-by: Amp <amp@ampcode.com>
2025-10-26 20:29:10 -07:00
Steve Yegge
6271b521b4 bd-162: Add database integrity checks with oracle review fixes
- Added validatePreExport to prevent data loss
- Added checkDuplicateIDs to detect corruption
- Added checkOrphanedDeps to find orphaned dependencies (both sides)
- Added validatePostImport to ensure imports don't lose data
- CRITICAL FIX: Removed post-pull export that clobbered fresh JSONL
- Conservative checks when JSONL is unreadable
- Efficient COUNT(*) SQL path instead of loading all issues
- Comprehensive test coverage including edge cases
2025-10-26 20:17:48 -07:00
Steve Yegge
b855c444d4 Enable errcheck linter and fix all production code warnings
- Enabled errcheck linter (previously disabled)
- Set tests: false in .golangci.yml to focus on production code
- Fixed 27 errcheck warnings using Go best practices:
  * Database resources: defer func() { _ = rows.Close() }()
  * Transaction rollbacks: defer func() { _ = tx.Rollback() }()
  * Best-effort closers: _ = store.Close(), _ = client.Close()
  * File writes: proper error checking on Close()
  * Interactive input: handle EOF gracefully
  * File ops: ignore ENOENT on os.Remove()
- All tests pass
- Closes bd-58

Amp-Thread-ID: https://ampcode.com/threads/T-57c9afd3-9adf-40c2-8be7-3e493d200361
Co-authored-by: Amp <amp@ampcode.com>
2025-10-25 18:44:38 -07:00
Steve Yegge
de03466da9 Fix bd-143: Prevent daemon auto-sync from wiping out issues.jsonl with empty database
- Added safety check to exportToJSONLWithStore (daemon path)
- Refuses to export 0 issues over non-empty JSONL file
- Added --force flag to override safety check when intentional
- Added test coverage for empty database export protection
- Prevents data loss when daemon has wrong/empty database

Amp-Thread-ID: https://ampcode.com/threads/T-de18e0ad-bd17-46ec-994b-0581e257dcde
Co-authored-by: Amp <amp@ampcode.com>
2025-10-25 16:36:18 -07:00
Steve Yegge
9a370b5b3c Fix gosec security warnings (bd-57)
- Changed file permissions from 0644 → 0600 for JSONL exports and config files
- Changed directory permissions from 0755 → 0750 in all test code
- Updated .golangci.yml with proper exclusions for false positives
- Reduced gosec warnings from 102 to 22 (all remaining are acceptable)

Closes bd-57

Amp-Thread-ID: https://ampcode.com/threads/T-f754d957-9e42-4e74-861e-57235c7e6436
Co-authored-by: Amp <amp@ampcode.com>
2025-10-25 13:50:32 -07:00
Steve Yegge
c59db1a798 fix: Resolve 11 errcheck linter violations to unblock CI (bd-91)
Fixed all unchecked error returns in production code:
- os.Remove() calls in cleanup paths
- cmd.Wait() in goroutines
- fmt.Fprintf() writes
- Type assertions with proper ok checks

Reduces linter issues from 99 to 88. CI should now pass linting.
2025-10-24 11:59:11 -07:00
Steve Yegge
e9729abd73 Fix bd-120: Fix nil pointer crash in export command when daemon is running
Amp-Thread-ID: https://ampcode.com/threads/T-f6d324a9-aa24-4cf8-9962-7391602c8c91
Co-authored-by: Amp <amp@ampcode.com>
2025-10-17 17:40:16 -07:00
Steve Yegge
3c8e4d78e5 Add label management and import collision detection
- Implement 'bd label' command with add/remove/list subcommands
- Add import --dry-run and --resolve-collisions for safe merges
- Support label filtering in 'bd list' and 'bd create'
- Update AGENTS.md with collision handling workflow
- Extends bd-224 (data consistency) and bd-84 (import reliability)
2025-10-15 17:19:16 -07:00
Matt Wilkie
e2703d3d9b Merge branch 'main' into win-defender-mitigation
# Conflicts:
#	.beads/issues.jsonl
#	README.md
2025-10-15 06:32:44 -07:00
Matt Wilkie
6e3498115f feat: refactor export file writing to avoid Windows Defender false positives
- Replace direct file creation with atomic temp file + rename pattern
- Add path validation to prevent writing to sensitive system directories
- Set proper file permissions (0644) on exported files
- Maintain backward compatibility and JSONL export functionality
- Reduce ransomware heuristic triggers through safer file operations
2025-10-14 13:51:34 -07:00
Steve Yegge
92759710de Fix race condition in dirty issue tracking (bd-52, bd-53)
Fix critical TOCTOU bug where concurrent operations could lose dirty
issue tracking, causing data loss in incremental exports. Also fixes
bug where export with filters would incorrectly clear all dirty issues.

The Problem:
1. GetDirtyIssues() returns [bd-1, bd-2]
2. Concurrent CRUD marks bd-3 dirty
3. Export writes bd-1, bd-2
4. ClearDirtyIssues() deletes ALL (including bd-3)
5. Result: bd-3 never gets exported!

The Fix:
- Add ClearDirtyIssuesByID() that only clears specific issue IDs
- Track which issues were actually exported
- Clear only those specific IDs, not all dirty issues
- Fixes both race condition and filter export bug

Changes:
- internal/storage/sqlite/dirty.go:
  * Add ClearDirtyIssuesByID() method
  * Add warning to ClearDirtyIssues() about race condition
- internal/storage/storage.go:
  * Add ClearDirtyIssuesByID to interface
- cmd/bd/main.go:
  * Update auto-flush to use ClearDirtyIssuesByID()
- cmd/bd/export.go:
  * Track exported issue IDs
  * Use ClearDirtyIssuesByID() instead of ClearDirtyIssues()

Testing:
- Created test-1, test-2, test-3 (all dirty)
- Updated test-2 to in_progress
- Exported with --status open filter (exports only test-1, test-3)
- Verified only test-2 remains dirty ✓
- All existing tests pass ✓

Impact:
- Race condition eliminated - concurrent operations are safe
- Export with filters now works correctly
- No data loss from competing writes

Closes bd-52, bd-53

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 00:29:23 -07:00
Steve Yegge
bafb2801c5 Implement incremental JSONL export with dirty issue tracking
Optimize auto-flush by tracking which issues have changed instead of
exporting the entire database on every flush. For large projects with
1000+ issues, this provides significant performance improvements.

Changes:
- Add dirty_issues table to schema with issue_id and marked_at columns
- Implement dirty tracking functions in new dirty.go file:
  * MarkIssueDirty() - Mark single issue as needing export
  * MarkIssuesDirty() - Batch mark multiple issues efficiently
  * GetDirtyIssues() - Query which issues need export
  * ClearDirtyIssues() - Clear tracking after successful export
  * GetDirtyIssueCount() - Monitor dirty issue count
- Update all CRUD operations to mark affected issues as dirty:
  * CreateIssue, UpdateIssue, DeleteIssue
  * AddDependency, RemoveDependency (marks both issues)
  * AddLabel, RemoveLabel, AddEvent
- Modify export to support incremental mode:
  * Add --incremental flag to export only dirty issues
  * Used by auto-flush for performance
  * Full export still available without flag
- Add Storage interface methods for dirty tracking

Performance impact: With incremental export, large databases only write
changed issues instead of regenerating entire JSONL file on every
auto-flush.

Closes bd-39

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 00:17:23 -07:00
Steve Yegge
a8a90e074e Add ID space partitioning and improve auto-flush reliability
Three improvements to beads:

1. ID space partitioning (closes bd-24)
   - Add --id flag to 'bd create' for explicit ID assignment
   - Validates format: prefix-number (e.g., worker1-100)
   - Enables parallel agents to partition ID space and avoid conflicts
   - Storage layer already supported this, just wired up CLI

2. Auto-flush failure tracking (closes bd-38)
   - Track consecutive flush failures with counter and last error
   - Show prominent red warning after 3+ consecutive failures
   - Reset counter on successful flush
   - Users get clear guidance to run manual export if needed

3. Manual export cancels auto-flush timer
   - Add clearAutoFlushState() helper function
   - bd export now cancels pending auto-flush and clears dirty flag
   - Prevents redundant exports when user manually exports
   - Also resets failure counter on successful manual export

Documentation updated in README.md and CLAUDE.md with --id flag examples.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 23:31:51 -07:00
Steve Yegge
3440899850 Fix code review issues bd-19 through bd-23
- bd-19: Fix import zero-value field handling using JSON presence detection
- bd-20: Add --strict flag for dependency import failures
- bd-21: Simplify getNextID SQL query (reduce params from 4 to 2)
- bd-22: Add validation/warning for malformed issue IDs
- bd-23: Optimize export dependency queries (fix N+1 problem)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 15:30:11 -07:00
Steve Yegge
8d3bbbcc52 Implement bd-10: Export/import dependencies in JSONL
Added dependency support to export/import workflow:

Changes:
- Added GetDependencyRecords() to Storage interface to get raw dependency records
- Extended Issue struct with Dependencies field (omitempty for backward compat)
- Modified export.go to populate dependencies for each issue
- Modified import.go to process dependencies in second pass after all issues exist
- All tests pass

Benefits:
- JSONL is now self-contained with full dependency information
- Enables proper collision resolution in future (bd-12+)
- Idempotent imports: existing dependencies are not duplicated
- Forward references handled: dependencies created after all issues exist

Example output:
{
  "id": "bd-10",
  "title": "...",
  "dependencies": [{
    "issue_id": "bd-10",
    "depends_on_id": "bd-9",
    "type": "parent-child",
    "created_at": "...",
    "created_by": "stevey"
  }]
}

Closes bd-10

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 15:08:05 -07:00
Steve Yegge
87ed7c8793 Polish for open-source release
Major improvements to code quality, documentation, and CI:

Code Quality:
- Add golangci-lint configuration with 13 linters
- Fix unchecked error returns in export/import/init
- Refactor duplicate scanIssues code
- Add package comments for all packages
- Add const block comments for exported constants
- Configure errcheck to allow idiomatic defer patterns

Documentation:
- Add comprehensive CONTRIBUTING.md with setup, testing, and workflow
- Fix QUICKSTART.md binary name references (beads → bd)
- Correct default database path documentation

CI/CD:
- Add GitHub Actions workflow for tests and linting
- Enable race detection and coverage reporting
- Automated quality checks on all PRs

All tests passing. Lint issues reduced from 117 to 103 (remaining are
idiomatic patterns and test code). Ready for open-source release.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 09:41:29 -07:00
Steve Yegge
15afb5ad17 Implement JSONL export/import and shift to text-first architecture
This is a fundamental architectural shift from binary SQLite to JSONL as
the source of truth for git workflows.

## New Features

- `bd export --format=jsonl` - Export issues to JSON Lines format
- `bd import` - Import issues from JSONL (create new, update existing)
- `--skip-existing` flag for import to only create new issues

## Architecture Change

**Before:** Binary SQLite database committed to git
**After:** JSONL text files as source of truth, SQLite as ephemeral cache

Benefits:
- Git-friendly text format with clean diffs
- AI-resolvable merge conflicts (append-only is 95% conflict-free)
- Human-readable issue tracking in git
- No binary merge conflicts

## Documentation

- Updated README with JSONL-first workflow and git hooks
- Added TEXT_FORMATS.md analyzing JSONL vs CSV vs binary
- Updated GIT_WORKFLOW.md with historical context
- .gitignore now excludes *.db, includes .beads/*.jsonl

## Implementation Details

- Export sorts issues by ID for consistent diffs
- Import handles both creates and updates atomically
- Proper handling of pointer fields (EstimatedMinutes)
- All tests passing

## Breaking Changes

- Database files (*.db) should now be gitignored
- Use export/import workflow for git collaboration
- Git hooks recommended for automation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-12 01:17:50 -07:00