beads/docs/contributor-workflow-analysis.md

Beads Contributor Workflow Analysis

Date: 2025-11-03 Context: Design discussion on how to handle beads issues in PR/OSS contribution workflows

The Problem (from #207)

When contributing to OSS projects with beads installed:

• Git hooks automatically commit contributor's personal planning to PRs
• Contributor's experimental musings pollute the upstream project's issue tracker
• No clear ownership/permission model for external contributors
• Difficult to keep beads changes out of commits

Core tension: Beads is great for team planning (shared namespace), but breaks down for OSS contributions (hierarchical gatekeeping).

Key Insights from Discussion

Beads as "Moving Frontier"

Beads is not a traditional issue tracker. It captures the active working set - the sliding window of issues currently under attention:

• Work moves fast with AI agents (10x-50x acceleration)
• Completed work fades quickly (95% never revisited, should be pruned aggressively)
• Future work is mostly blocked (small frontier of ready tasks)
• The frontier is bounded by team size (dozens to hundreds of issues, not thousands)

Design principle: Beads should focus on the "what's next" cloud, not long-term planning or historical archive.

The Git Ledger is Fundamental

Beads achieves reliability despite being unreliable (merge conflicts, sync issues, data staleness) through:

A. Git is the ledger and immutable backstop for forensics B. AI is the ultimate arbiter and problem-solver when things go wrong

Any solution that removes the git ledger (e.g., gitignored contributor files) breaks this model entirely.

Requirements for Contributors

Contributors need:

• Git-backed persistence (multi-clone sync, forensics, AI repair)
• Isolated planning space (don't pollute upstream)
• Ability to propose selected issues upstream
• Support for multiple workers across multiple clones of the same repo

Proposed Solutions

Idea 1: Fork-Aware Hooks + Two-File System

Structure:

# Upstream repo
.beads/
  beads.jsonl          # Canonical frontier (committed)
  .gitignore           # Ignores local.jsonl

# Contributor's fork
.beads/
  beads.jsonl          # Synced from upstream (read-only)
  local.jsonl          # Contributor planning (committed to fork)
  beads.db             # Hydrated from both

Detection: Check for upstream remote to distinguish fork from canonical repo

Workflow:

# In fork
$ bd add "Experiment"        # → local.jsonl (committed to fork)
$ bd sync                    # → Pulls upstream's beads.jsonl
$ bd show                    # → Shows both
$ bd propose bd-a3f8e9       # → Moves issue to beads.jsonl for PR

Pros:

• Git ledger preserved (local.jsonl committed to fork)
• Multi-clone sync works
• Upstream .gitignore prevents pollution

Cons:

• Fork detection doesn't help teams using branches (most common workflow)
• Two files to manage
• Requires discipline to use bd propose

Idea 2: Ownership Metadata + Smart PR Filtering

Structure:

{"id":"bd-123","owner":"upstream","title":"Canonical issue",...}
{"id":"bd-456","owner":"stevey","title":"My planning",...}

Workflow:

$ bd add "Experiment"        # → Creates with owner="stevey"
$ bd propose bd-456          # → Changes owner to "upstream"
$ bd clean-pr                # → Filters commit to only upstream-owned issues
$ git push                   # → PR contains only proposed issues

Pros:

• Single file (simpler)
• Works with any git workflow (branch, fork, etc)
• Git ledger fully preserved

Cons:

• Requires discipline to run bd clean-pr
• Clean commit is awkward (temporarily removing data)
• Merge conflicts if upstream and contributor both modify beads.jsonl

Idea 3: Branch-Scoped Databases

Track which issues belong to which branch, filter at PR time.

Implementation: Similar to #2 but uses labels/metadata to track branch instead of owner.

Challenge: Complex with multiple feature branches, requires tracking branch scope.

Idea 4: Separate Planning Repo (Most Isolated)

Structure:

# Main project repos (many)
~/projects/beads/.beads/beads.jsonl
~/projects/foo/.beads/beads.jsonl

# Single planning repo (one)
~/.beads-planning/.beads/beads.jsonl

# Configuration links them
~/projects/beads/.beads/config.toml:
  planning_repo = "~/.beads-planning"

Workflow:

$ cd ~/projects/beads
$ bd add "My idea"           # → Commits to ~/.beads-planning/
$ bd show                    # → Shows beads canonical + my planning
$ bd propose bd-a3f8e9       # → Exports to beads repo for PR

Pros:

• Complete isolation (separate git histories, zero PR pollution risk)
• Git ledger fully preserved (both repos tracked)
• Multi-clone works perfectly (clone both repos)
• No special filtering/detection needed
• Scales better: One planning repo for all projects

Cons:

• Two repos to manage
• Less obvious for new users (where's my planning?)

Analysis: Fork vs Clone vs Branch

Clone: Local copy of a repo (git clone <url>)

• origin remote points to source
• Push directly to origin (if you have write access)

Fork: Server-side copy on GitHub

• For contributors without write access
• origin → your fork, upstream → original repo
• Push to fork, then PR from fork → upstream

Branch: Feature branches in same repo

• Most common for teams with write access
• Push to same repo, PR from branch → main

Key insight: Branches are universal, forks are only for external contributors. Most teams work on branches in a shared repo.

Current Thinking: Idea 4 is Cleanest

After analysis, separate planning repo (#4) is likely the best solution because:

1. Only solution that truly prevents PR pollution (separate git histories)
2. Git ledger fully preserved (both repos tracked)
3. Multi-clone works perfectly (just clone both)
4. No complex filtering/detection needed (simple config)
5. Better scaling: One planning repo across all projects you contribute to

The "managing two repos" concern is actually an advantage: your planning is centralized and project-agnostic.

Open Questions

About the Workflow

1. Where does PR pollution actually happen?

   • Scenario A: Feature branch → upstream/main includes all beads changes from that branch?
   • Scenario B: Something else?

2. Multi-clone usage pattern:

   • Multiple clones on different machines?
   • All push/pull to same remote?
   • Workers coordinate via git sync?
   • PRs created from feature branches?

About Implementation

1. Proposed issue IDs: When moving issue from planning → canonical, keep same ID? (Hash-based IDs are globally unique)

2. Upstream acceptance sync: If upstream accepts/modifies a proposal, how to sync back to contributor?

   • bd sync detects accepted proposals
   • Moves from planning repo to project's canonical beads.jsonl

3. Multiple projects: One planning repo for all projects you contribute to, or one per project?

4. Backwards compatibility: Single-user projects unchanged (single beads.jsonl)

5. Discovery: How do users discover this feature? Auto-detect and prompt?

Next Steps

Need to clarify:

1. User's actual multi-clone workflow (understand the real use case)
2. Where exactly PR pollution occurs (branch vs fork workflow)
3. Which solution best fits the "git ledger + multi-clone" requirements
4. Whether centralized planning repo (#4) or per-project isolation (#1/#2) is preferred

Design Principles to Preserve

From the conversation, these are non-negotiable:

• Git as ledger: Everything must be git-tracked for forensics and AI repair
• Moving frontier: Focus on active work, aggressively prune completed work
• Multi-clone sync: Workers across clones must coordinate via git
• Small databases: Keep beads.jsonl small enough for agents to read (<25k per repo, see below)
• Simple defaults: Don't break single-user workflows
• Explicit over implicit: Clear boundaries between personal and canonical

JSONL Size Bounds with Multi-Repo

Critical clarification: The <25k limit applies per-repo, not to total hydrated size.

The Rule

Per-repo limit: Each individual JSONL file should stay <25k (roughly 100-200 issues depending on metadata).

Why per-repo, not total:

1. Git operations: Each repo is independently versioned. Git performance depends on per-file size, not aggregate.
2. AI readability: Agents read JSONLs for forensics/repair. Reading one 20k file is easy; reading the union of 10 files is still manageable.
3. Bounded growth: Total size naturally bounded by number of repos (typically N=1-3, rarely >10).
4. Pruning granularity: Completed work is pruned per-repo, keeping each repo's frontier small.

Example Scenarios

Primary	Planning	Team Shared	Total Hydrated	Valid?
20k	-	-	20k	✅ Single-repo, well under limit
20k	15k	-	35k	✅ Each repo <25k (per-repo rule)
20k	15k	18k	53k	✅ Each repo <25k (per-repo rule)
30k	15k	-	45k	❌ Primary exceeds 25k
20k	28k	-	48k	❌ Planning exceeds 25k

Rationale: Why 25k?

Agent context limits: AI agents have finite context windows. A 25k JSONL file is:

• ~100-200 issues with metadata
• ~500-1000 lines of JSON
• Comfortably fits in GPT-4 context (128k tokens)
• Small enough to read/parse in <500ms

Moving frontier principle: Beads tracks active work, not historical archive. With aggressive pruning:

• Completed issues get compacted/archived
• Blocked work stays dormant
• Only ready + in-progress issues are "hot"
• Typical frontier: 50-100 issues per repo

Monitoring Size with Multi-Repo

Per-repo monitoring:

# Check each repo's JSONL size
$ wc -c .beads/beads.jsonl
20480 .beads/beads.jsonl

$ wc -c ~/.beads-planning/beads.jsonl
15360 ~/.beads-planning/beads.jsonl

# Total hydrated size (informational, not a hard limit)
$ expr 20480 + 15360
35840

Automated check:

// Check all configured repos
for _, repo := range cfg.Repos.All() {
    jsonlPath := filepath.Join(repo, "beads.jsonl")
    size, _ := getFileSize(jsonlPath)
    if size > 25*1024 {  // 25k
        log.Warnf("Repo %s exceeds 25k limit: %d bytes", repo, size)
    }
}

Pruning Strategy with Multi-Repo

Each repo should be pruned independently:

# Prune completed work from primary repo
$ bd compact --repo . --older-than 30d

# Prune experimental planning repo
$ bd compact --repo ~/.beads-planning --older-than 7d

# Shared team planning (longer retention)
$ bd compact --repo ~/team-shared/.beads --older-than 90d

Different repos can have different retention policies based on their role.

Total Size Soft Limit (Guideline Only)

While per-repo limit is the hard rule, consider total hydrated size for performance:

Guideline: Keep total hydrated size <100k for optimal performance.

Why 100k total:

• SQLite hydration: Parsing 100k JSON still fast (<1s)
• Agent queries: Dependency graphs with 300-500 total issues remain tractable
• Memory footprint: In-memory SQLite comfortably handles 500 issues

If total exceeds 100k:

• Not a hard error, but performance may degrade
• Consider pruning completed work more aggressively
• Evaluate if all repos are still needed
• Check if any repos should be archived/removed

Summary

Limit Type	Value	Enforcement
Per-repo (hard limit)	<25k	⚠️ Warn if exceeded, agents may struggle
Total hydrated (guideline)	<100k	ℹ️ Informational, affects performance
Typical usage	20k-50k total	✅ Expected range for active development

Bottom line: Monitor per-repo size (<25k each). Total size naturally bounded by N repos × 25k.

---

Decision: Separate Repos (Solution #4)

Date: 2025-11-03 (continued discussion)

Why Separate Repos

After consideration, Solution #4 (Separate Planning Repos) is the chosen approach:

Key Rationale

1. Beads as a Separate Channel: Beads is fundamentally a separate communication channel that happens to use git/VCS for persistence, not a git-centric tool. It should work with any VCS (jujutsu, sapling, mercurial, etc.).

2. VCS-Agnostic Design: Solution #1 (fork detection) is too git-centric and wouldn't work with other version control systems. Separate repos work regardless of VCS.

3. Maximum Flexibility: Supports multiple workflows and personas:

   • OSS contributor with personal planning
   • Multi-phase development (different beads DBs for different stages)
   • Multiple personas (architect, implementer, reviewer)
   • Team vs personal planning separation

4. Zero PR Pollution Risk: Completely separate git histories guarantee no accidental pollution of upstream projects.

5. Proven Pain Point: Experience shows that accidental bulk commits (100k issues) can be catastrophic and traumatic to recover from. Complete isolation is worth the complexity.

Core Architecture Principles

1. Multi-Repo Support (N ≥ 1)

Configuration should support N repos, including N=1 for backward compatibility:

When N=1 (default), this is the current single-repo workflow - no changes needed. When N≥2, multiple repos are hydrated together.

# .beads/config.toml

# Default mode: single repo (backwards compatible)
mode = "single"

# Multi-repo mode
[repos]
  # Primary repo: where canonical issues live
  primary = "."

  # Additional repos to hydrate into the database
  additional = [
    "~/.beads-planning",           # Personal planning across all projects
    "~/.beads-work/phase1",        # Architecting phase
    "~/.beads-work/phase2",        # Implementation phase
    "~/team-shared/.beads",        # Shared team planning
  ]

# Routing: where do new issues go?
[routing]
  mode = "auto"  # auto | explicit
  default = "~/.beads-planning"    # Default for `bd add`

  # Auto-detection: based on user permissions
  [routing.auto]
    maintainer = "."                   # If maintainer, use primary
    contributor = "~/.beads-planning"  # Otherwise use planning repo

2. Hydration Model

On bd show, bd list, etc., the database hydrates from multiple sources:

beads.db ← [
  ./.beads/beads.jsonl           (primary, read-write if maintainer)
  ~/.beads-planning/beads.jsonl   (personal, read-write)
  ~/team-shared/.beads/beads.jsonl (shared, read-write if team member)
]

Metadata tracking:

{
  "id": "bd-a3f8e9",
  "title": "Add dark mode",
  "source_repo": "~/.beads-planning",  # Which repo owns this issue
  "visibility": "local",               # local | proposed | canonical
  ...
}

3. Visibility States

Issues can be in different states of visibility:

• local: Personal planning, only in planning repo
• proposed: Exported for upstream consideration (staged for PR)
• canonical: In the primary repo (upstream accepted it)

4. VCS-Agnostic Operations

Beads should not assume git. Core operations:

• Sync: bd sync should work with git, jj, hg, sl, etc.
• Ledger: Each repo uses whatever VCS it's under (or none)
• Transport: Issues move between repos via export/import, not git-specific operations

Workflow Examples

Use Case 1: OSS Contributor

# One-time setup
$ mkdir ~/.beads-planning
$ cd ~/.beads-planning
$ git init && bd init

# Contributing to upstream project
$ cd ~/projects/some-oss-project
$ bd config --add-repo ~/.beads-planning --routing contributor

# Work
$ bd add "Explore dark mode implementation"
# → Goes to ~/.beads-planning/beads.jsonl
# → Commits to planning repo (git tracked, forensic trail)

$ bd show
# → Shows upstream's canonical issues (read-only)
# → Shows my planning issues (read-write)

$ bd work bd-a3f8e9
$ bd status bd-a3f8e9 in-progress

# Ready to propose
$ bd propose bd-a3f8e9 --target upstream
# → Exports issue from planning repo
# → Creates issue in ./beads/beads.jsonl (staged for PR)
# → Marks as visibility="proposed" in planning repo

$ git add .beads/beads.jsonl
$ git commit -m "Propose: Add dark mode"
$ git push origin feature-branch
# → PR contains only the proposed issue, not all my planning

Use Case 2: Multi-Phase Development

# Setup phases
$ mkdir -p ~/.beads-work/{architecture,implementation,testing}
$ for dir in ~/.beads-work/*; do (cd $dir && git init && bd init); done

# Configure project
$ cd ~/my-big-project
$ bd config --add-repo ~/.beads-work/architecture
$ bd config --add-repo ~/.beads-work/implementation
$ bd config --add-repo ~/.beads-work/testing

# Architecture phase
$ bd add "Design authentication system" --repo ~/.beads-work/architecture
$ bd show --repo ~/.beads-work/architecture
# → Only architecture issues

# Implementation phase (later)
$ bd add "Implement JWT validation" --repo ~/.beads-work/implementation

# View all phases
$ bd show
# → Shows all issues from all configured repos

Use Case 3: Multiple Contributors on Same Project

# Team member Alice (maintainer)
$ cd ~/project
$ bd add "Fix bug in parser"
# → Goes to ./beads/beads.jsonl (she's maintainer)
# → Commits to project repo

# Team member Bob (contributor)
$ cd ~/project
$ bd add "Explore performance optimization"
# → Goes to ~/.beads-planning/beads.jsonl (he's contributor)
# → Does NOT pollute project repo

$ bd show
# → Sees Alice's canonical issue
# → Sees his own planning

$ bd propose bd-xyz
# → Proposes to Alice's canonical repo

Implementation Outline

Phase 1: Core Multi-Repo Support

Commands:

bd config --add-repo <path>         # Add a repo to hydration
bd config --remove-repo <path>      # Remove a repo
bd config --list-repos              # Show all configured repos
bd config --routing <mode>          # Set routing: single|auto|explicit

Config schema:

[repos]
primary = "."
additional = ["path1", "path2", ...]

[routing]
default = "path"  # Where `bd add` goes by default
mode = "auto"     # auto | explicit

Database changes:

• Add source_repo field to issues
• Hydration layer reads from multiple JSONLs
• Writes go to correct JSONL based on source_repo

Phase 2: Proposal Flow

Commands:

bd propose <issue-id> [--target <repo>]   # Move issue to target repo
bd withdraw <issue-id>                    # Un-propose (move back)
bd accept <issue-id>                      # Maintainer accepts proposal

States:

• visibility: local → Personal planning
• visibility: proposed → Staged for PR
• visibility: canonical → Accepted by upstream

Phase 3: Routing Rules

Auto-detection:

• Detect if user is maintainer (git config, permissions)
• Auto-route to primary vs planning repo

Config-based routing (no new schema fields):

[routing]
mode = "auto"  # auto | explicit
default = "~/.beads-planning"  # Fallback for contributors

# Auto-detection rules
[routing.auto]
maintainer = "."  # If user is maintainer, use primary repo
contributor = "~/.beads-planning"  # Otherwise use planning repo

Explicit routing via CLI flag:

# Override auto-detection for specific issues
bd add "Design system" --repo ~/.beads-work/architecture

Discovered issue inheritance:

• Issues with parent_id automatically inherit parent's source_repo
• Keeps related work co-located

Phase 4: VCS-Agnostic Sync

Sync operations:

• Detect VCS type per repo (git, jj, hg, sl)
• Use appropriate sync commands
• Fall back to manual sync if no VCS

Example:

$ bd sync
# Auto-detects:
# - . is git → runs git pull
# - ~/.beads-planning is jj → runs jj git fetch && jj rebase
# - ~/other is hg → runs hg pull && hg update

Migration Path

Existing Users (Single Repo)

No changes required. Current workflow continues to work:

$ bd add "Task"
# → .beads/beads.jsonl (as before)

Library Consumers (Go/TypeScript)

Critical for projects like VC that use beads as a library.

Backward Compatibility (No Changes Required)

Your existing code continues to work unchanged. The storage layer automatically reads .beads/config.toml if present:

// Before multi-repo (v0.17.3)
store, err := beadsLib.NewSQLiteStorage(".beads/vc.db")

// After multi-repo (v0.18.0+) - EXACT SAME CODE
store, err := beadsLib.NewSQLiteStorage(".beads/vc.db")
// If .beads/config.toml exists, additional repos are auto-hydrated
// If .beads/config.toml doesn't exist, single-repo mode (backward compatible)

What happens automatically:

1. Storage layer checks for .beads/config.toml
2. If found: Reads repos.additional, hydrates from all configured repos
3. If not found: Single-repo mode (current behavior)
4. Your code doesn't need to know which mode is active

Explicit Multi-Repo Configuration (Optional)

If you need to override config.toml or configure repos programmatically:

// Explicit multi-repo configuration
cfg := beadsLib.Config{
    Primary:    ".beads/vc.db",
    Additional: []string{
        filepath.ExpandUser("~/.beads-planning"),
        filepath.ExpandUser("~/team-shared/.beads"),
    },
}
store, err := beadsLib.NewStorageWithConfig(cfg)

When to use explicit configuration:

• Testing: Override config for test isolation
• Dynamic repos: Add repos based on runtime conditions
• No config file: Programmatic setup without .beads/config.toml

When to Use Multi-Repo vs Single-Repo

Single-repo (default, recommended for most library consumers):

// VC executor managing its own database
store, err := beadsLib.NewSQLiteStorage(".beads/vc.db")
// Stays single-repo by default, no config.toml needed

Multi-repo (opt-in for specific use cases):

• Team planning: VC executor needs to see team-wide issues from shared repo
• Multi-phase dev: Different repos for architecture, implementation, testing phases
• Personal planning: User wants to track personal experiments separate from VC's canonical DB

Example: VC with team planning:

# .beads/config.toml
[repos]
primary = "."
additional = ["~/team-shared/.beads"]

[routing]
default = "."  # VC-generated issues go to primary

// VC executor code (unchanged)
store, err := beadsLib.NewSQLiteStorage(".beads/vc.db")

// GetReadyWork() now returns issues from:
// - .beads/vc.db (VC-generated issues)
// - ~/team-shared/.beads (team planning)
ready, err := store.GetReadyWork(ctx)

Migration Checklist for Library Consumers

1. Test with config.toml: Create .beads/config.toml, verify auto-hydration works
2. Verify performance: Ensure multi-repo hydration meets your latency requirements (see Performance section)
3. Update exclusive locks: If using locks, decide if you need per-repo or all-repo locking (see Exclusive Lock Protocol section)
4. Review routing: Ensure auto-generated issues (e.g., VC's discovered:blocker) go to correct repo
5. Test backward compat: Verify code works with and without config.toml

API Compatibility Matrix

API Call	v0.17.3 (single-repo)	v0.18.0+ (multi-repo)	Breaking?
NewSQLiteStorage(path)	✅ Single repo	✅ Auto-detects config	❌ No
GetReadyWork()	✅ Returns from single DB	✅ Returns from all repos	❌ No
CreateIssue()	✅ Writes to single DB	✅ Writes to primary (or routing config)	❌ No
UpdateIssue()	✅ Updates in single DB	✅ Updates in source repo	❌ No
Exclusive locks	✅ Locks single DB	✅ Locks per-repo	❌ No

Summary: Zero breaking changes. Multi-repo is transparent to library consumers.

Opting Into Multi-Repo (CLI Users)

# Create planning repo
$ mkdir ~/.beads-planning && cd ~/.beads-planning
$ git init && bd init

# Link to project
$ cd ~/my-project
$ bd config --add-repo ~/.beads-planning
$ bd config --routing auto  # Auto-detect maintainer vs contributor

# Optionally migrate existing issues
$ bd migrate --move-to ~/.beads-planning --filter "author=me"

Teams Adopting Beads

# Maintainer sets up project
$ cd ~/team-project
$ bd init
$ git add .beads/ && git commit -m "Initialize beads"

# Contributors clone and configure
$ git clone team-project
$ cd team-project
$ mkdir ~/.beads-planning && cd ~/.beads-planning
$ git init && bd init
$ cd ~/team-project
$ bd config --add-repo ~/.beads-planning --routing contributor

Self-Hosting Projects (VC, Internal Tools, Pet Projects)

Important: The multi-repo design is primarily for OSS contributors making PRs to upstream projects. Self-hosting projects have different needs.

What is Self-Hosting?

Projects that use beads to build themselves:

• VC (VibeCoder): Uses beads to track development of VC itself
• Internal tools: Company tools that track their own roadmap
• Pet projects: Personal projects with beads-based planning

Key difference from OSS contribution:

• No upstream/downstream distinction (you ARE the project)
• Direct commit access (no PR workflow)
• Often have automated executors/agents
• Bootstrap/early phase stability matters

Default Recommendation: Stay Single-Repo

For most self-hosting projects, single-repo is the right choice:

# Simple, stable, proven
$ cd ~/my-project
$ bd init
$ bd create "Task" -p 1
# → .beads/beads.jsonl (committed to project repo)

Why single-repo for self-hosting:

• ✅ Simpler: No config, no routing decisions, no multi-repo complexity
• ✅ Proven: Current architecture, battle-tested
• ✅ Sufficient: All issues live with the project they describe
• ✅ Stable: No hydration overhead, no cross-repo coordination

When to Adopt Multi-Repo

Multi-repo makes sense for self-hosting projects only in specific scenarios:

Scenario 1: Team Planning Separation

Your project has multiple developers with different permission levels:

# .beads/config.toml
[repos]
primary = "."  # Canonical project issues (maintainers only)
additional = ["~/team-shared/.beads"]  # Team planning (all contributors)

Scenario 2: Multi-Phase Development

Your project uses distinct phases (architecture → implementation → testing):

# .beads/config.toml
[repos]
primary = "."  # Current active work
additional = [
  "~/.beads-work/architecture",  # Design decisions
  "~/.beads-work/implementation", # Implementation backlog
]

Scenario 3: Experimental Work Isolation

You want to keep experimental ideas separate from canonical roadmap:

# .beads/config.toml
[repos]
primary = "."  # Committed roadmap
additional = ["~/.beads-experiments"]  # Experimental ideas

Automated Executors with Multi-Repo

Critical for projects like VC with automated agents.

Default behavior (recommended):

// Executor sees ONLY primary repo (canonical work)
store, err := beadsLib.NewSQLiteStorage(".beads/vc.db")
// No config.toml = single-repo mode
ready, err := store.GetReadyWork(ctx)  // Only canonical issues

With multi-repo (opt-in):

# .beads/config.toml
[repos]
primary = "."
additional = ["~/team-shared/.beads"]

[routing]
default = "."  # Executor-created issues stay in primary

// Executor code (unchanged)
store, err := beadsLib.NewSQLiteStorage(".beads/vc.db")
// Auto-reads config.toml, hydrates from both repos
ready, err := store.GetReadyWork(ctx)
// Returns issues from primary + team-shared

// When executor creates discovered issues:
discovered := &Issue{Title: "Found blocker", ...}
store.CreateIssue(discovered)
// → Goes to primary repo (routing.default = ".")

Recommendation for executors: Stay single-repo unless you have a clear team coordination need.

Bootstrap Phase Considerations

If your project is in early/bootstrap phase (like VC), extra caution:

1. Prioritize stability: Multi-repo adds complexity. Delay until proven need.
2. Test thoroughly: If adopting multi-repo, test with small repos first.
3. Monitor performance: Ensure executor polling loops stay sub-second (see Performance section).
4. Plan rollback: Keep single-repo workflow working so you can revert if needed.

Bootstrap-phase checklist:

• Do you have multiple developers with different permissions? → Maybe multi-repo
• Do you have team planning separate from executor roadmap? → Maybe multi-repo
• Are you solo or small team with unified planning? → Stay single-repo
• Is executor stability critical right now? → Stay single-repo
• Can you afford multi-repo testing/debugging time? → If no, stay single-repo

Migration Path for Self-Hosting Projects

From single-repo to multi-repo (when ready):

# Step 1: Create planning repo
$ mkdir ~/.beads-planning && cd ~/.beads-planning
$ git init && bd init

# Step 2: Configure multi-repo (test mode)
$ cd ~/my-project
$ bd config --add-repo ~/.beads-planning --routing auto

# Step 3: Test with small workload
$ bd create "Test issue" --repo ~/.beads-planning
$ bd show  # Verify hydration works
$ bd ready  # Verify queries work

# Step 4: Verify executor compatibility
# - Run executor with multi-repo config
# - Check GetReadyWork() latency (<100ms)
# - Verify discovered issues route correctly

# Step 5: Migrate planning issues (optional)
$ bd migrate --move-to ~/.beads-planning --filter "label=experimental"

Rollback (if needed):

# Remove config.toml to revert to single-repo
$ rm .beads/config.toml
$ bd show  # Back to single-repo mode

Summary: Self-Hosting Decision Tree

Is your project self-hosting? (building itself with beads)
├─ YES
│  ├─ Solo developer or unified team?
│  │  └─ Stay single-repo (simple, stable)
│  ├─ Multiple developers, different permissions?
│  │  └─ Consider multi-repo (team planning separation)
│  ├─ Multi-phase development (arch → impl → test)?
│  │  └─ Consider multi-repo (phase isolation)
│  ├─ Bootstrap/early phase?
│  │  └─ Stay single-repo (stability > flexibility)
│  └─ Automated executor?
│     └─ Stay single-repo unless team coordination needed
└─ NO (OSS contributor)
   └─ Use multi-repo (planning repo separate from upstream)

Bottom line for self-hosting: Default to single-repo. Only adopt multi-repo when you have a proven, specific need.

Design Decisions (Resolved)

1. Namespace Collisions: Option B (Global Uniqueness)

Decision: Use globally unique hash-based IDs that include timestamp + random component.

Rationale (from VC feedback):

• Option C (allow collisions) breaks dependency references: bd dep add bd-a3f8e9 bd-b7c2d1 becomes ambiguous
• Need to support cross-repo dependencies without repo-scoped namespacing
• Hash should be: hash(title + description + timestamp_ms + random_4bytes)
• Collision probability: ~1 in 10^12 (acceptable)

2. Cross-Repo Dependencies: Yes, Fully Supported

Decision: Dependencies work transparently across all repos.

Implementation:

• Hydrated database contains all issues from all repos
• Dependencies stored by ID only (no repo qualifier needed)
• bd ready checks dependency graph across all repos
• Writes route back to correct JSONL via source_repo metadata

3. Routing Mechanism: Config-Based, No Schema Changes

Decision: Use config-based routing + explicit --repo flag. No new schema fields.

Rationale:

• IssueType already exists and is used semantically (bug, feature, task, epic, chore)
• Labels are used semantically by VC (discovered:blocker, no-auto-claim)
• Routing is a storage concern, not issue metadata
• Simpler: auto-detect maintainer vs contributor from config
• Discovered issues inherit parent's source_repo automatically

4. Performance: Smart Caching with File Mtime Tracking

Decision: SQLite DB is the cache, JSONLs are source of truth.

Implementation:

type MultiRepoStorage struct {
    repos      []RepoConfig
    db         *sql.DB
    repoMtimes map[string]time.Time  // Track file modification times
}

func (s *MultiRepoStorage) GetReadyWork(ctx) ([]Issue, error) {
    // Fast path: check if ANY JSONL changed
    needSync := false
    for repo, jsonlPath := range s.jsonlPaths() {
        currentMtime := stat(jsonlPath).ModTime()
        if currentMtime.After(s.repoMtimes[repo]) {
            needSync = true
            s.repoMtimes[repo] = currentMtime
        }
    }

    // Only re-hydrate if something changed
    if needSync {
        s.rehydrate()  // Expensive but rare
    }

    // Query is fast (in-memory SQLite)
    return s.db.Query("SELECT * FROM issues WHERE ...")
}

Rationale: VC's polling loop (every 5-10 seconds) requires sub-second queries. File stat is microseconds, re-parsing only when needed.

Performance Benchmarks and Targets

Critical for library consumers (VC) with automated polling.

Performance Targets

Based on VC's polling loop requirements (every 5-10 seconds):

Operation	Target	Rationale
File stat (per repo)	<1ms	Checking mtime of N JSONLs must be negligible
Hydration (full re-parse)	<500ms	Only happens when JSONL changes, rare in polling loop
Query (from cached DB)	<10ms	Common case: no JSONL changes, pure SQLite query
Total GetReadyWork()	<100ms	VC's hard requirement for responsive executor

Scale Testing Targets

Test at multiple repo counts to ensure scaling:

Repo Count	File Stat Total	Hydration (worst case)	Query (cached)	Total (cached)
N=1 (baseline)	<1ms	<200ms	<5ms	<10ms
N=3 (typical)	<3ms	<500ms	<10ms	<20ms
N=10 (edge case)	<10ms	<2s	<50ms	<100ms

Assumptions:

• JSONL size: <25k per repo (see Design Principles)
• SQLite: In-memory mode (:memory: or file::memory:?cache=shared)
• Cached case: No JSONL changes since last hydration (99% of polling loops)

Benchmark Suite (To Be Implemented)

// benchmark/multi_repo_test.go

func BenchmarkFileStatOverhead(b *testing.B) {
    // Test: Stat N JSONL files
    // Target: <1ms per repo
}

func BenchmarkHydrationN1(b *testing.B) {
    // Test: Full hydration from 1 JSONL (20k file)
    // Target: <200ms
}

func BenchmarkHydrationN3(b *testing.B) {
    // Test: Full hydration from 3 JSONLs (20k each)
    // Target: <500ms
}

func BenchmarkHydrationN10(b *testing.B) {
    // Test: Full hydration from 10 JSONLs (20k each)
    // Target: <2s
}

func BenchmarkQueryCached(b *testing.B) {
    // Test: GetReadyWork() with no JSONL changes
    // Target: <10ms
}

func BenchmarkGetReadyWorkN3(b *testing.B) {
    // Test: Realistic polling loop (3 repos, cached)
    // Target: <20ms total
}

Performance Optimization Notes

If benchmarks fail to meet targets, optimization strategies:

1. Parallel file stats: Use goroutines to stat N JSONLs concurrently
2. Incremental hydration: Only re-parse changed repos, merge into DB
3. Smarter caching: Hash-based cache invalidation (mtime + file size)
4. SQLite tuning: PRAGMA synchronous = OFF, PRAGMA journal_mode = MEMORY
5. Lazy hydration: Defer hydration until first query after mtime change

Status

Benchmarks: ⏳ Not implemented yet (tracked in bd-wta) Targets: ✅ Documented above Validation: ⏳ Pending implementation

Next steps:

1. Implement benchmark suite in benchmark/multi_repo_test.go
2. Run benchmarks on realistic workloads (VC-sized DBs)
3. Document results in this section
4. File optimization issues if targets not met

5. Visibility Field: Optional, Backward Compatible

Decision: Add visibility as optional field, defaults to "canonical" if missing.

Schema:

type Issue struct {
    // ... existing fields ...
    Visibility *string `json:"visibility,omitempty"`  // nil = canonical
}

States:

• local: Personal planning only
• proposed: Staged for upstream PR
• canonical: Accepted by upstream (or default for existing issues)

Orthogonality: Visibility and Status are independent:

• status: in_progress, visibility: local → Working on personal planning
• status: open, visibility: proposed → Proposed to upstream, awaiting review

6. Library API Stability: Transparent Hydration

Decision: Hybrid approach - transparent by default, explicit opt-in available.

Backward Compatible:

// Existing code keeps working - reads config.toml automatically
store, err := beadsLib.NewSQLiteStorage(".beads/vc.db")

Explicit Override:

// Library consumers can override config
cfg := beadsLib.Config{
    Primary: ".beads/vc.db",
    Additional: []string{"~/.beads-planning"},
}
store, err := beadsLib.NewStorageWithConfig(cfg)

7. ACID Guarantees: Per-Repo File Locking

Decision: Use file-based locks per JSONL, atomic within single repo.

Implementation:

func (s *Storage) UpdateIssue(issue Issue) error {
    sourceRepo := issue.SourceRepo

    // Lock that repo's JSONL
    lock := flock(sourceRepo + "/beads.jsonl.lock")
    defer lock.Unlock()

    // Read-modify-write
    issues := s.readJSONL(sourceRepo)
    issues.Update(issue)
    s.writeJSONL(sourceRepo, issues)

    // Update in-memory DB
    s.db.Update(issue)
}

Limitation: Cross-repo transactions are NOT atomic (acceptable, rare use case).

Compatibility with Exclusive Lock Protocol

The per-repo file locking (Decision #7) is fully compatible with the existing exclusive lock protocol (see EXCLUSIVE_LOCK.md).

How they work together:

1. Exclusive locks are daemon-level: The .beads/.exclusive-lock prevents the bd daemon from operating on a specific database
2. File locks are operation-level: Per-JSONL file locks (flock) ensure atomic read-modify-write for individual operations
3. Different scopes, complementary purposes:
   • Exclusive lock: "This entire database is off-limits to the daemon"
   • File lock: "This specific JSONL is being modified right now"

Multi-repo behavior:

With multi-repo configuration, each repo can have its own exclusive lock:

# VC executor locks its primary database
.beads/.exclusive-lock           # Locks primary repo operations

# Planning repo can be locked independently
~/.beads-planning/.exclusive-lock  # Locks planning repo operations

When both are active:

• If primary repo is locked: Daemon skips all operations on primary, but can still sync planning repo
• If planning repo is locked: Daemon skips planning repo, but can still sync primary
• If both locked: Daemon skips entire multi-repo workspace

No migration needed for library consumers:

Existing VC code (v0.17.3+) using exclusive locks will continue to work:

// VC's existing lock acquisition
lock, err := types.NewExclusiveLock("vc-executor", "1.0.0")
lockPath := filepath.Join(".beads", ".exclusive-lock")
os.WriteFile(lockPath, data, 0644)

// Works the same with multi-repo:
// - Locks .beads/ (primary repo)
// - Daemon skips primary, can still sync ~/.beads-planning if configured

Atomic multi-repo locking:

If a library consumer needs to lock all repos atomically:

// Lock all configured repos
repos := []string{".beads", filepath.ExpandUser("~/.beads-planning")}
for _, repo := range repos {
    lockPath := filepath.Join(repo, ".exclusive-lock")
    os.WriteFile(lockPath, lockData, 0644)
}
defer func() {
    for _, repo := range repos {
        os.Remove(filepath.Join(repo, ".exclusive-lock"))
    }
}()

// Now daemon skips all repos until locks released

Summary: No breaking changes. Exclusive locks work per-repo in multi-repo configs, preventing daemon interference at repo granularity.

Key Learnings from VC Feedback

The VC project (VibeCoder) provided critical feedback as a real downstream consumer that uses beads as a library. Key insights:

1. Two Consumer Models

Beads has two distinct consumer types:

• CLI users: Use bd commands directly
• Library consumers: Use beadsLib in Go/TypeScript/etc. (like VC)

Multi-repo must work transparently for both.

2. Performance is Critical for Automation

VC's executor polls GetReadyWork() every 5-10 seconds. Multi-repo hydration must:

• Use smart caching (file mtime tracking)
• Avoid re-parsing JSONLs on every query
• Keep queries sub-second (ideally <100ms)

3. Special Labels Must Work Across Repos

VC uses semantic labels that must work regardless of repo:

• discovered:blocker - Auto-generated blocker issues (priority boost)
• discovered:related - Auto-generated related work
• no-auto-claim - Prevent executor from claiming
• baseline-failure - Self-healing baseline failures

These are semantic labels, not routing labels. Don't overload labels for routing.

4. Discovered Issues Routing

When VC's analysis phase auto-creates issues with discovered:blocker label, they should:

• Inherit parent's source_repo automatically
• Stay co-located with related work
• Not require manual routing decisions

5. Library API Stability is Non-Negotiable

VC's code uses beadsLib.NewSQLiteStorage(). Must not break. Solution:

• Read .beads/config.toml automatically (transparent)
• Provide NewStorageWithConfig() for explicit override
• Hydration happens at storage layer, invisible to library consumers

Remaining Open Questions

1. Sync semantics: When upstream accepts a proposed issue and modifies it, how to sync back?

   • Option A: Mark as "accepted" in planning repo, keep both copies
   • Option B: Delete from planning repo (it's now canonical)
   • Option C: Keep in planning repo but mark as read-only mirror

2. Discovery: How do users learn about this feature?

   • Auto-prompt when detecting fork/contributor status?
   • Docs + examples?
   • bd init --contributor wizard?

3. Metadata fields: Should source_repo be exposed in JSON export, or keep it internal to storage layer?

4. Proposed issue lifecycle: What happens to proposed issues after PR is merged/rejected?

   • Auto-delete from planning repo?
   • Mark as "accepted" or "rejected"?
   • Manual cleanup via bd withdraw?

Success Metrics

How we'll know this works:

1. Zero pollution: No contributor planning issues accidentally merged upstream
2. Multi-clone sync: Workers on different machines see consistent state (via VCS sync)
3. Flexibility: Users can configure for their workflow (personas, phases, etc.)
4. Backwards compat: Existing single-repo users unaffected
5. VCS-agnostic: Works with git, jj, hg, sl, or no VCS

Next Actions

Suggested epics/issues to create (can be done in follow-up session):

1. Epic: Multi-repo hydration layer

   • Design schema for source_repo metadata
   • Implement config parsing for repos.additional
   • Build hydration logic (read from N JSONLs)
   • Build write routing (write to correct JSONL)

2. Epic: Proposal workflow

   • Implement bd propose command
   • Implement bd withdraw command
   • Implement bd accept command (maintainer only)
   • Design visibility state machine

3. Epic: Auto-routing

   • Detect maintainer vs contributor status
   • Implement routing rules (label, priority, custom)
   • Make bd add route to correct repo

4. Epic: VCS-agnostic sync

   • Detect VCS type per repo
   • Implement sync adapters (git, jj, hg, sl)
   • Handle mixed-VCS multi-repo configs

5. Epic: Migration and onboarding

   • Write migration guide
   • Implement bd migrate command
   • Create init wizards for common scenarios
   • Update documentation

---

Summary and Next Steps

This document represents the design evolution for multi-repo support in beads, driven by:

1. Original problem (GitHub #207): Contributors' personal planning pollutes upstream PRs
2. Core insight: Beads is a separate communication channel that happens to use VCS
3. VC feedback: Real-world library consumer with specific performance and API stability needs

Final Architecture

Solution #4 (Separate Repos) with these refinements:

• N ≥ 1 repos: Single repo (N=1) is default, multi-repo is opt-in
• VCS-agnostic: Works with git, jj, hg, sapling, or no VCS
• Config-based routing: No schema changes, auto-detect maintainer vs contributor
• Smart caching: File mtime tracking, SQLite DB as cache layer
• Transparent hydration: Library API remains stable, config-driven
• Global namespace: Hash-based IDs with timestamp + random for uniqueness
• Cross-repo dependencies: Fully supported, transparent to users
• Discovered issues: Inherit parent's source_repo automatically

Why This Design Wins

1. Zero PR pollution: Separate git histories = impossible to accidentally merge planning
2. Git ledger preserved: All repos are VCS-tracked, full forensic capability
3. Maximum flexibility: Supports OSS contributors, multi-phase dev, multi-persona workflows
4. Backward compatible: Existing single-repo users unchanged
5. Performance: Sub-second queries even with polling loops
6. Library-friendly: Transparent to downstream consumers like VC

Related Documents

• Original issue: GitHub #207
• VC feedback: ./vc-feedback-on-multi-repo.md
• Implementation tracking: TBD (epics to be created)

Status

Design: ✅ Complete (pending resolution of open questions) Implementation: ⏳ Not started Target: TBD

Last updated: 2025-11-03