Files
beads/cmd/bd/sync_external_test.go
Peter Chanthamynavong 1561374c04 feat(sync): pull-first sync with 3-way merge (#918)
* feat(sync): implement pull-first synchronization strategy

- Add --pull-first flag and logic to sync command
- Introduce 3-way merge stub for issue synchronization
- Add concurrent edit tests for the pull-first flow

Ensures local changes are reconciled with remote updates before pushing to prevent data loss.

* feat(sync): implement 3-way merge and state tracking

- Implement 3-way merge algorithm for issue synchronization
- Add base state storage to track changes between syncs
- Add comprehensive tests for merge logic and persistence

Ensures data consistency and prevents data loss during concurrent
issue updates.

* feat(sync): implement field-level conflict merging

- Implement field-level merge logic for issue conflicts
- Add unit tests for field-level merge strategies

Reduces manual intervention by automatically resolving overlapping updates at the field level.

* refactor(sync): simplify sync flow by removing ZFC checks

The previous sync implementation relied on Zero-False-Convergence (ZFC)
staleness checks which are redundant following the transition to
structural 3-way merging. This legacy logic added complexity and
maintenance overhead without providing additional safety.

This commit introduces a streamlined sync pipeline:
- Remove ZFC staleness validation from primary sync flow
- Update safety documentation to reflect current merge strategy
- Eliminate deprecated unit tests associated with ZFC logic

These changes reduce codebase complexity while maintaining data
integrity through the robust structural 3-way merge implementation.

* feat(sync): default to pull-first sync workflow

- Set pull-first as the primary synchronization workflow
- Refactor core sync logic for better maintainability
- Update concurrent edit tests to validate 3-way merge logic

Reduces merge conflicts by ensuring local state is current before pushing changes.

* refactor(sync): clean up lint issues in merge code

- Remove unused error return from MergeIssues (never returned error)
- Use _ prefix for unused _base parameter in mergeFieldLevel
- Update callers to not expect error from MergeIssues
- Keep nolint:gosec for trusted internal file path

* test(sync): add mode compatibility and upgrade safety tests

Add tests addressing Steve's PR #918 review concerns:

- TestSyncBranchModeWithPullFirst: Verifies sync-branch config
  storage and git branch creation work with pull-first
- TestExternalBeadsDirWithPullFirst: Verifies external BEADS_DIR
  detection and pullFromExternalBeadsRepo
- TestUpgradeFromOldSync: Validates upgrade safety when
  sync_base.jsonl doesn't exist (first sync after upgrade)
- TestMergeIssuesWithBaseState: Comprehensive 3-way merge cases
- TestLabelUnionMerge: Verifies labels use union (no data loss)

Key upgrade behavior validated:
- base=nil (no sync_base.jsonl) safely handles all cases
- Local-only issues kept (StrategyLocal)
- Remote-only issues kept (StrategyRemote)
- Overlapping issues merged (LWW scalars, union labels)

* fix(sync): report line numbers for malformed JSON

Problem:
- JSON decoding errors when loading sync base state lacked line numbers
- Difficult to identify location of syntax errors in large state files

Solution:
- Include line number reporting in JSON decoder errors during state loading
- Add regression tests for malformed sync base file scenarios

Impact:
- Users receive actionable feedback for corrupted state files
- Faster troubleshooting of manual configuration errors

* fix(sync): warn on large clock skew during sync

Problem:
- Unsynchronized clocks between systems could lead to silent merge errors
- No mechanism existed to alert users of significant timestamp drift

Solution:
- Implement clock skew detection during sync merge
- Log a warning when large timestamp differences are found
- Add comprehensive unit tests for skew reporting

Impact:
- Users are alerted to potential synchronization risks
- Easier debugging of time-related merge issues

* fix(sync): defer state update until remote push succeeds

Problem:
- Base state updated before confirming remote push completion
- Failed pushes resulted in inconsistent local state tracking

Solution:
- Defer base state update until after the remote push succeeds

Impact:
- Ensures local state accurately reflects remote repository status
- Prevents state desynchronization during network or push failures

* fix(sync): prevent concurrent sync operations

Problem:
- Multiple sync processes could run simultaneously
- Overlapping operations risk data corruption and race conditions

Solution:
- Implement file-based locking using gofrs/flock
- Add integration tests to verify locking behavior

Impact:
- Guarantees execution of a single sync process at a time
- Eliminates potential for data inconsistency during sync

* docs: document sync architecture and merge model

- Detail the 3-way merge model logic
- Describe the core synchronization architecture principles

* fix(lint): explicitly ignore lock.Unlock return value

errcheck linter flagged bare defer lock.Unlock() calls. Wrap in
anonymous function with explicit _ assignment to acknowledge
intentional ignore of unlock errors during cleanup.

* fix(lint): add sync_merge.go to G304 exclusions

The loadBaseState and saveBaseState functions use file paths derived
from trusted internal sources (beadsDir parameter from config). Add
to existing G304 exclusion list for safe JSONL file operations.

* feat(sync): integrate sync-branch into pull-first flow

When sync.branch is configured, doPullFirstSync now:
- Calls PullFromSyncBranch before merge
- Calls CommitToSyncBranch after export

This ensures sync-branch mode uses the correct branch for
pull/push operations.

* test(sync): add E2E tests for sync-branch and external BEADS_DIR

Adds comprehensive end-to-end tests:
- TestSyncBranchE2E: verifies pull→merge→commit flow with remote changes
- TestExternalBeadsDirE2E: verifies sync with separate beads repository
- TestExternalBeadsDirDetection: edge cases for repo detection
- TestCommitToExternalBeadsRepo: commit handling

* refactor(sync): remove unused rollbackJSONLFromGit

Function was defined but never called. Pull-first flow saves base
state after successful push, making this safety net unnecessary.

* test(sync): add export-only mode E2E test

Add TestExportOnlySync to cover --no-pull flag which was the only
untested sync mode. This completes full mode coverage:

- Normal (pull-first): sync_test.go, sync_merge_test.go
- Sync-branch: sync_modes_test.go:TestSyncBranchE2E (PR#918)
- External BEADS_DIR: sync_external_test.go (PR#918)
- From-main: sync_branch_priority_test.go
- Local-only: sync_local_only_test.go
- Export-only: sync_modes_test.go:TestExportOnlySync (this commit)

Refs: #911

* docs(sync): add sync modes reference section

Document all 6 sync modes with triggers, flows, and use cases.
Include mode selection decision tree and test coverage matrix.

Co-authored-by: Claude <noreply@anthropic.com>

* test(sync): upgrade sync-branch E2E tests to bare repo

- Replace mocked repository with real bare repo setup
- Implement multi-machine simulation in sync tests
- Refactor test logic to handle distributed states

Coverage: sync-branch end-to-end scenarios

* test(sync): add daemon sync-branch E2E tests

- Implement E2E tests for daemon sync-branch flow
- Add test cases for force-overwrite scenarios

Coverage: daemon sync-branch workflow in cmd/bd

* docs(sync): document sync-branch paths and E2E architecture

- Describe sync-branch CLI and Daemon execution flow
- Document the end-to-end test architecture

* build(nix): update vendorHash for gofrs/flock dependency

New dependency added for file-based sync locking changes the
Go module checksum.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-07 21:27:20 -08:00

366 lines
12 KiB
Go

package main
import (
"context"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"github.com/steveyegge/beads/internal/git"
)
// TestExternalBeadsDirE2E tests the full external BEADS_DIR flow.
// This is an end-to-end regression test for PR#918.
//
// When BEADS_DIR points to a separate git repository (external mode),
// sync operations should work correctly:
// 1. Changes are committed to the external beads repo (not the project repo)
// 2. Pulls from the external repo bring in remote changes
// 3. The merge algorithm works correctly across repo boundaries
func TestExternalBeadsDirE2E(t *testing.T) {
ctx := context.Background()
// Store original working directory
originalWd, err := os.Getwd()
if err != nil {
t.Fatalf("failed to get original working directory: %v", err)
}
// Setup: Create the main project repo
projectDir := t.TempDir()
if err := setupGitRepoInDir(t, projectDir); err != nil {
t.Fatalf("failed to setup project repo: %v", err)
}
// Setup: Create a separate external beads repo
// Resolve symlinks to avoid macOS /var -> /private/var issues
externalDir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatalf("eval symlinks failed: %v", err)
}
if err := setupGitRepoInDir(t, externalDir); err != nil {
t.Fatalf("failed to setup external repo: %v", err)
}
// Create .beads directory in external repo
externalBeadsDir := filepath.Join(externalDir, ".beads")
if err := os.MkdirAll(externalBeadsDir, 0755); err != nil {
t.Fatalf("failed to create external .beads dir: %v", err)
}
// Create issues.jsonl in external beads repo with initial issue
jsonlPath := filepath.Join(externalBeadsDir, "issues.jsonl")
issue1 := `{"id":"ext-1","title":"External Issue 1","status":"open","issue_type":"task","priority":2,"created_at":"2025-01-01T00:00:00Z","updated_at":"2025-01-01T00:00:00Z"}`
if err := os.WriteFile(jsonlPath, []byte(issue1+"\n"), 0644); err != nil {
t.Fatalf("write external JSONL failed: %v", err)
}
// Commit initial beads files in external repo
runGitInDir(t, externalDir, "add", ".beads")
runGitInDir(t, externalDir, "commit", "-m", "initial beads setup")
t.Log("✓ External beads repo initialized with issue ext-1")
// Change to project directory (simulating user's project)
if err := os.Chdir(projectDir); err != nil {
t.Fatalf("chdir to project failed: %v", err)
}
defer func() { _ = os.Chdir(originalWd) }()
// Reset git caches after directory change
git.ResetCaches()
// Test 1: isExternalBeadsDir should detect external repo
if !isExternalBeadsDir(ctx, externalBeadsDir) {
t.Error("isExternalBeadsDir should return true for external beads dir")
}
t.Log("✓ External beads dir correctly detected")
// Test 2: getRepoRootFromPath should correctly identify external repo root
repoRoot, err := getRepoRootFromPath(ctx, externalBeadsDir)
if err != nil {
t.Fatalf("getRepoRootFromPath failed: %v", err)
}
// Normalize paths for comparison
resolvedExternal, _ := filepath.EvalSymlinks(externalDir)
if repoRoot != resolvedExternal {
t.Errorf("getRepoRootFromPath = %q, want %q", repoRoot, resolvedExternal)
}
t.Logf("✓ getRepoRootFromPath correctly identifies external repo: %s", repoRoot)
// Test 3: pullFromExternalBeadsRepo should handle no-remote gracefully
err = pullFromExternalBeadsRepo(ctx, externalBeadsDir)
if err != nil {
t.Errorf("pullFromExternalBeadsRepo should handle no-remote: %v", err)
}
t.Log("✓ Pull from external beads repo handled no-remote correctly")
// Test 4: Create new issue and commit to external repo
issue2 := `{"id":"ext-2","title":"External Issue 2","status":"open","issue_type":"task","priority":2,"created_at":"2025-01-02T00:00:00Z","updated_at":"2025-01-02T00:00:00Z"}`
combinedContent := issue1 + "\n" + issue2 + "\n"
if err := os.WriteFile(jsonlPath, []byte(combinedContent), 0644); err != nil {
t.Fatalf("write updated JSONL failed: %v", err)
}
// Use commitToExternalBeadsRepo (don't push since no real remote)
committed, err := commitToExternalBeadsRepo(ctx, externalBeadsDir, "add ext-2", false)
if err != nil {
t.Fatalf("commitToExternalBeadsRepo failed: %v", err)
}
if !committed {
t.Error("expected commit to succeed for new issue")
}
t.Log("✓ Successfully committed issue ext-2 to external beads repo")
// Test 5: Verify commit was made in external repo (not project repo)
// Check external repo has the commit
logOutput := getGitOutputInDir(t, externalDir, "log", "--oneline", "-1")
if !strings.Contains(logOutput, "add ext-2") {
t.Errorf("external repo should have commit, got: %s", logOutput)
}
t.Log("✓ Commit correctly made in external repo")
// Test 6: Verify project repo is unchanged
projectLogOutput := getGitOutputInDir(t, projectDir, "log", "--oneline", "-1")
if strings.Contains(projectLogOutput, "add ext-2") {
t.Error("project repo should not have beads commit")
}
t.Log("✓ Project repo correctly unchanged")
// Test 7: Verify JSONL content is correct
content, err := os.ReadFile(jsonlPath)
if err != nil {
t.Fatalf("failed to read JSONL: %v", err)
}
contentStr := string(content)
if !strings.Contains(contentStr, "ext-1") || !strings.Contains(contentStr, "ext-2") {
t.Errorf("JSONL should contain both issues, got: %s", contentStr)
}
t.Log("✓ JSONL contains both issues")
t.Log("✓ External BEADS_DIR E2E test completed")
}
// TestExternalBeadsDirDetection tests various edge cases for external beads dir detection.
func TestExternalBeadsDirDetection(t *testing.T) {
ctx := context.Background()
// Store original working directory
originalWd, err := os.Getwd()
if err != nil {
t.Fatalf("failed to get original working directory: %v", err)
}
t.Run("same repo returns false", func(t *testing.T) {
// Setup a single repo
repoDir := t.TempDir()
if err := setupGitRepoInDir(t, repoDir); err != nil {
t.Fatalf("setup failed: %v", err)
}
beadsDir := filepath.Join(repoDir, ".beads")
if err := os.MkdirAll(beadsDir, 0755); err != nil {
t.Fatalf("mkdir failed: %v", err)
}
// Change to repo dir
if err := os.Chdir(repoDir); err != nil {
t.Fatalf("chdir failed: %v", err)
}
defer func() { _ = os.Chdir(originalWd) }()
git.ResetCaches()
// Same repo should return false
if isExternalBeadsDir(ctx, beadsDir) {
t.Error("isExternalBeadsDir should return false for same repo")
}
})
t.Run("different repo returns true", func(t *testing.T) {
// Setup two separate repos
projectDir := t.TempDir()
if err := setupGitRepoInDir(t, projectDir); err != nil {
t.Fatalf("setup project failed: %v", err)
}
externalDir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatalf("eval symlinks failed: %v", err)
}
if err := setupGitRepoInDir(t, externalDir); err != nil {
t.Fatalf("setup external failed: %v", err)
}
beadsDir := filepath.Join(externalDir, ".beads")
if err := os.MkdirAll(beadsDir, 0755); err != nil {
t.Fatalf("mkdir failed: %v", err)
}
// Change to project dir
if err := os.Chdir(projectDir); err != nil {
t.Fatalf("chdir failed: %v", err)
}
defer func() { _ = os.Chdir(originalWd) }()
git.ResetCaches()
// Different repo should return true
if !isExternalBeadsDir(ctx, beadsDir) {
t.Error("isExternalBeadsDir should return true for different repo")
}
})
t.Run("non-git directory returns false", func(t *testing.T) {
// Setup a repo for cwd
repoDir := t.TempDir()
if err := setupGitRepoInDir(t, repoDir); err != nil {
t.Fatalf("setup failed: %v", err)
}
// Non-git beads dir
nonGitDir := t.TempDir()
beadsDir := filepath.Join(nonGitDir, ".beads")
if err := os.MkdirAll(beadsDir, 0755); err != nil {
t.Fatalf("mkdir failed: %v", err)
}
// Change to repo dir
if err := os.Chdir(repoDir); err != nil {
t.Fatalf("chdir failed: %v", err)
}
defer func() { _ = os.Chdir(originalWd) }()
git.ResetCaches()
// Non-git dir should return false (can't determine, assume local)
if isExternalBeadsDir(ctx, beadsDir) {
t.Error("isExternalBeadsDir should return false for non-git directory")
}
})
}
// TestCommitToExternalBeadsRepo tests the external repo commit function.
func TestCommitToExternalBeadsRepo(t *testing.T) {
ctx := context.Background()
t.Run("commits changes to external repo", func(t *testing.T) {
// Setup external repo
externalDir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatalf("eval symlinks failed: %v", err)
}
if err := setupGitRepoInDir(t, externalDir); err != nil {
t.Fatalf("setup failed: %v", err)
}
beadsDir := filepath.Join(externalDir, ".beads")
if err := os.MkdirAll(beadsDir, 0755); err != nil {
t.Fatalf("mkdir failed: %v", err)
}
// Write initial JSONL
jsonlPath := filepath.Join(beadsDir, "issues.jsonl")
if err := os.WriteFile(jsonlPath, []byte(`{"id":"test-1"}`+"\n"), 0644); err != nil {
t.Fatalf("write failed: %v", err)
}
// Commit
committed, err := commitToExternalBeadsRepo(ctx, beadsDir, "test commit", false)
if err != nil {
t.Fatalf("commit failed: %v", err)
}
if !committed {
t.Error("expected commit to succeed")
}
// Verify commit exists
logOutput := getGitOutputInDir(t, externalDir, "log", "--oneline", "-1")
if !strings.Contains(logOutput, "test commit") {
t.Errorf("commit not found in log: %s", logOutput)
}
})
t.Run("returns false when no changes", func(t *testing.T) {
// Setup external repo
externalDir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatalf("eval symlinks failed: %v", err)
}
if err := setupGitRepoInDir(t, externalDir); err != nil {
t.Fatalf("setup failed: %v", err)
}
beadsDir := filepath.Join(externalDir, ".beads")
if err := os.MkdirAll(beadsDir, 0755); err != nil {
t.Fatalf("mkdir failed: %v", err)
}
// Write and commit JSONL
jsonlPath := filepath.Join(beadsDir, "issues.jsonl")
if err := os.WriteFile(jsonlPath, []byte(`{"id":"test-1"}`+"\n"), 0644); err != nil {
t.Fatalf("write failed: %v", err)
}
runGitInDir(t, externalDir, "add", ".beads")
runGitInDir(t, externalDir, "commit", "-m", "initial")
// Try to commit again with no changes
committed, err := commitToExternalBeadsRepo(ctx, beadsDir, "no changes", false)
if err != nil {
t.Fatalf("commit failed: %v", err)
}
if committed {
t.Error("expected no commit when no changes")
}
})
}
// Helper: Setup git repo in a specific directory (doesn't change cwd)
func setupGitRepoInDir(t *testing.T, dir string) error {
t.Helper()
// Initialize git repo
if err := exec.Command("git", "-C", dir, "init", "--initial-branch=main").Run(); err != nil {
return err
}
// Configure git
_ = exec.Command("git", "-C", dir, "config", "user.email", "test@test.com").Run()
_ = exec.Command("git", "-C", dir, "config", "user.name", "Test User").Run()
// Create initial commit
readmePath := filepath.Join(dir, "README.md")
if err := os.WriteFile(readmePath, []byte("# Test Repo\n"), 0644); err != nil {
return err
}
if err := exec.Command("git", "-C", dir, "add", ".").Run(); err != nil {
return err
}
if err := exec.Command("git", "-C", dir, "commit", "-m", "initial commit").Run(); err != nil {
return err
}
return nil
}
// Helper: Run git command in a specific directory
func runGitInDir(t *testing.T, dir string, args ...string) {
t.Helper()
cmdArgs := append([]string{"-C", dir}, args...)
cmd := exec.Command("git", cmdArgs...)
output, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("git %v failed: %v\n%s", args, err, output)
}
}
// Helper: Get git output from a specific directory
func getGitOutputInDir(t *testing.T, dir string, args ...string) string {
t.Helper()
cmdArgs := append([]string{"-C", dir}, args...)
cmd := exec.Command("git", cmdArgs...)
output, err := cmd.Output()
if err != nil {
t.Fatalf("git %v failed: %v", args, err)
}
return string(output)
}