Adds operational improvements to the Dolt storage backend for increased
reliability in production environments with concurrent clients:
1. Lock retry with exponential backoff:
- Add LockRetries and LockRetryDelay config options
- Automatic retry on lock contention (default: 30 retries, ~6s window)
- Exponential backoff starting at 100ms
- Handles transient format version errors during manifest updates
2. Stale lock file cleanup:
- Detect and clean orphaned .dolt/noms/LOCK files on startup
- Prevents "database is read only" errors after crashes
- Only removes empty locks older than 5 seconds
3. Transient error detection:
- isTransientDoltError() detects retryable conditions
- isLockError() identifies lock contention scenarios
- cleanupStaleDoltLock() safely removes orphaned locks
These improvements address common issues in multi-process environments
where the Dolt embedded driver creates exclusive locks that persist
after unexpected termination.
Co-authored-by: upstream_syncer <matthew.baker@pihealth.ai>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
The Dolt storage was scanning created_at and updated_at directly into
time.Time fields, but SQLite stores these as TEXT strings. The Go SQLite
driver cannot automatically convert TEXT to time.Time.
Added parseTimeString() helper and fixed all scan functions:
- issues.go: scanIssue()
- dependencies.go: scanIssueRow()
- history.go: GetIssueHistory(), GetIssueAsOf()
- transaction.go: scanIssueTx()
Fixes bd-4dqmy
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add integration tests for the Gas Town hub+spokes topology where:
- Hub (mayor rig) runs dolt sql-server with remotesapi
- Spokes (crew clones) configure hub as their only peer
- Data flows through hub in star topology
Tests included:
- TestHubSpokes_MultiCloneSync: Basic two-spoke convergence
- TestHubSpokes_WorkDispatch: Hub dispatches work, spokes complete it
- TestHubSpokes_ThreeSpokesConverge: Three-spoke convergence test
Note: These tests follow the same pattern as peer_sync_integration_test.go
and document current behavior including the no common ancestor limitation
that affects cross-database sync.
Closes bd-phwci
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Improve the federation status command to show more comprehensive
information similar to `git remote -v` with health info:
- Show peer URLs alongside peer names
- Display pending local changes count (uncommitted)
- Test connectivity to each peer (via fetch)
- Track and display last sync time in metadata table
- Show reachability status with error messages on failure
Changes:
- cmd/bd/federation.go: Enhanced status output with URLs, connectivity
checks, pending changes, and last sync time
- internal/storage/dolt/federation.go: Added getLastSyncTime/setLastSyncTime
methods using metadata table, record sync time on successful sync
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Merge SQL user authentication with Emma federation sync implementation:
- Add federation_peers table for encrypted credential storage
- Add credentials.go with AES-256-GCM encryption, SHA-256 key derivation
- Extend FederatedStorage interface with credential methods
- Add --user, --password, --sovereignty flags to bd federation add-peer
- Integrate credentials into PushTo/PullFrom/Fetch via withPeerCredentials
- DOLT_REMOTE_USER/PASSWORD env vars protected by mutex for concurrency
Credentials automatically used when syncing with peers that have stored auth.
Continues: bd-wkumz.10, Closes: bd-4p67y
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add --federation-port and --remotesapi-port flags (default 3306/8080)
- Fix log file leak in server.go - track and close on Stop()
- Add BEADS_DOLT_PASSWORD env var for server mode authentication
- Update DSN to include password when set
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add --federation flag to bd daemon start that runs dolt sql-server
instead of the embedded driver. Enables multi-writer support and
exposes remotesapi on port 8080 for peer-to-peer push/pull.
Changes:
- Add --federation flag to daemon start command
- Create dolt server manager (internal/storage/dolt/server.go)
- Update DoltStore to support server mode via MySQL protocol
- Integrate server lifecycle into daemon (auto-start/stop)
- Add tests for server management and server mode connections
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add support for importing routes.jsonl and interactions.jsonl during
Dolt bootstrap. Previously only issues.jsonl was imported.
Changes:
- Add routes and interactions tables to Dolt schema
- Import routes before issues (no dependencies)
- Import interactions after issues (may reference issue_id)
- Reuse audit.Entry type instead of duplicating
- Add tests for multi-file bootstrap
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add versioned storage methods for incremental export support:
- GetChangesSinceExport: returns changes since a commit hash, with
NeedsFullExport flag for invalid/GC'd commits
- CommitExists: checks if a commit hash exists, supports short prefixes
Also fixes dolt_diff syntax for embedded driver (from_ref, to_ref, table).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add peer-to-peer synchronization for Dolt-backed beads databases:
- New FederatedStorage interface with PushTo, PullFrom, Fetch, ListRemotes,
RemoveRemote, and SyncStatus methods
- DoltStore implementation using DOLT_PUSH, DOLT_PULL, DOLT_FETCH
- Full bd federation command with subcommands:
- sync: bidirectional sync with conflict resolution (--strategy ours|theirs)
- status: show ahead/behind counts and conflict state
- add-peer/remove-peer/list-peers: manage federation remotes
- Comprehensive tests for all federation APIs
Closes: bd-wkumz.4
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add integration tests validating Dolt APIs needed for peer-to-peer
federation between Gas Towns:
- TestFederationDatabaseIsolation: verify separate towns have isolated DBs
- TestFederationVersionControlAPIs: branch, checkout, merge, commit
- TestFederationRemoteConfiguration: AddRemote accepts dolthub:// URLs
- TestFederationHistoryQueries: commit history, dolt_log, AS OF
Key finding: Production federation uses dolt sql-server remotesapi
(port 8080) for direct peer-to-peer push/pull - no DoltHub required.
Also fixes concurrent_test.go Merge() return value handling.
Refs: bd-wkumz (Federation epic), hq-ew1mbr.7 (prototype task)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(config): remove duplicate declarations between config.go and sync.go
Commit e82e15a8 created sync.go with typed constants (SyncMode,
ConflictStrategy, Sovereignty) but didn't remove the original untyped
constants from config.go that were added in 16f8c3d3. This caused
redeclaration errors preventing the project from building.
Changes:
- Remove duplicate SyncMode, ConflictStrategy, Sovereignty constants
from config.go (keep typed versions in sync.go)
- Remove duplicate GetSyncMode, GetConflictStrategy, GetSovereignty
functions from config.go (keep sync.go versions with warnings)
- Update SyncConfig, ConflictConfig, FederationConfig structs to use
typed fields instead of string
- Add IsSyncModeValid, IsConflictStrategyValid, IsSovereigntyValid
wrapper functions that use sync.go's validation maps
- Update cmd/bd/sync.go to use typed ConflictStrategy parameter
- Update tests to work with typed constants
* fix(dolt): handle Merge return values in concurrent test
* fix(test): add --repo flag to show_test.go to bypass auto-routing
The tests were failing because the create command was routing issues
to ~/.beads-planning instead of the test's temp directory. Adding
--repo . overrides auto-routing and creates issues in the test dir.
Extends Storage interface with Dolt-specific version control capabilities:
- New VersionedStorage interface in storage/versioned.go with:
- History queries: History(), AsOf(), Diff()
- Branch operations: Branch(), Merge(), CurrentBranch(), ListBranches()
- Commit operations: Commit(), GetCurrentCommit()
- Conflict resolution: GetConflicts(), ResolveConflicts()
- Helper types: HistoryEntry, DiffEntry, Conflict
- DoltStore implements VersionedStorage interface
- New CLI commands:
- bd history <id> - Show issue version history
- bd diff <from> <to> - Show changes between commits/branches
- bd branch [name] - List or create branches
- bd vc merge <branch> - Merge branch to current
- bd vc commit -m <msg> - Create a commit
- bd vc status - Show current branch/commit
- Added --as-of flag to bd show for time-travel queries
- IsVersioned() helper for graceful SQLite backend detection
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add DeleteBranch method to DoltStore for removing branches
- Update hookPostMergeDolt to clean up import branches after merge
- Completes hq-ew1mbr.9 git hook infrastructure
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add batch query optimization to avoid N+1 queries in scanIssueIDs
- Create GetIssuesByIDs helper to fetch multiple issues in single query
- Add scanIssueRow helper to scan issue data from rows iterator
- Add proper timeout contexts to all Dolt tests using testContext helper
The embedded Dolt driver is slow for repeated queries. Replacing N+1
GetIssue calls with a single IN clause query fixes the 30s+ timeouts
in TestDoltStoreDependencies, TestDoltStoreSearch, and
TestDoltStoreGetReadyWork.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use exec.LookPath instead of hardcoded path for Dolt detection
- Add test context with timeout to prevent tests from hanging
- Document known issues with embedded Dolt driver async operations
The embedded Dolt driver can hang on complex JOIN queries. This change
ensures tests fail gracefully with timeout rather than hanging.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
All time.Now() calls in the dolt storage layer now use time.Now().UTC()
to ensure consistent timezone handling. Previously, timestamps could be
stored with mixed timezone formats (UTC 'Z' vs local '+01:00'), causing
bv validation to fail when updated_at appeared earlier than created_at
in absolute time.
Files modified:
- transaction.go: CreateIssue, UpdateIssue, CloseIssue
- issues.go: CreateIssue, CreateIssues, UpdateIssue, CloseIssue, markDirty, manageClosedAt
- rename.go: UpdateIssueID (2 locations)
- events.go: AddIssueComment (2 locations)
- dirty.go: SetExportHash
- queries.go: Overdue filter, GetStaleIssues
Fixes: bd-84gw9
Co-authored-by: LoomDeBWiles <loomenwiles@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
The SQL formatting warnings (G201) are safe because:
- Placeholders only contain "?" markers for parameterized queries
- WHERE/SET clauses use validated column names with ? placeholders
- Refs are validated by validateRef() before use in AS OF queries
- LIMIT values are safe integers from filter.Limit
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase 2 of Dolt integration - enables runtime backend selection:
- Add --backend flag to bd init (sqlite|dolt)
- Create storage factory for backend instantiation
- Update daemon and main.go to use factory with config detection
- Update database discovery to find Dolt backends via metadata.json
- Fix Dolt schema init to split statements for MySQL compatibility
- Add ReadOnly mode to skip schema init for read-only commands
Usage: bd init --backend dolt --prefix myproject
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implements a complete Dolt storage backend that mirrors the SQLite implementation
with MySQL-compatible syntax and adds version control capabilities.
Key features:
- Full Storage interface implementation (~50 methods)
- Version control operations: commit, push, pull, branch, merge, checkout
- History queries via AS OF and dolt_history_* tables
- Cell-level merge instead of line-level JSONL merge
- SQL injection protection with input validation
Bug fixes applied during implementation:
- Added missing quality_score, work_type, source_system to scanIssue
- Fixed Status() to properly parse boolean staged column
- Added validation to CreateIssues (was missing in batch create)
- Made RenameDependencyPrefix transactional
- Expanded GetIssueHistory to return more complete data
Test coverage: 17 tests covering CRUD, dependencies, labels, search,
comments, events, statistics, and SQL injection protection.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
