beads/docs/contributor-workflow-analysis.md

Beads Contributor Workflow Analysis

Date: 2025-11-03 Context: Design discussion on how to handle beads issues in PR/OSS contribution workflows

The Problem (from #207)

When contributing to OSS projects with beads installed:

• Git hooks automatically commit contributor's personal planning to PRs
• Contributor's experimental musings pollute the upstream project's issue tracker
• No clear ownership/permission model for external contributors
• Difficult to keep beads changes out of commits

Core tension: Beads is great for team planning (shared namespace), but breaks down for OSS contributions (hierarchical gatekeeping).

Key Insights from Discussion

Beads as "Moving Frontier"

Beads is not a traditional issue tracker. It captures the active working set - the sliding window of issues currently under attention:

• Work moves fast with AI agents (10x-50x acceleration)
• Completed work fades quickly (95% never revisited, should be pruned aggressively)
• Future work is mostly blocked (small frontier of ready tasks)
• The frontier is bounded by team size (dozens to hundreds of issues, not thousands)

Design principle: Beads should focus on the "what's next" cloud, not long-term planning or historical archive.

The Git Ledger is Fundamental

Beads achieves reliability despite being unreliable (merge conflicts, sync issues, data staleness) through:

A. Git is the ledger and immutable backstop for forensics B. AI is the ultimate arbiter and problem-solver when things go wrong

Any solution that removes the git ledger (e.g., gitignored contributor files) breaks this model entirely.

Requirements for Contributors

Contributors need:

• Git-backed persistence (multi-clone sync, forensics, AI repair)
• Isolated planning space (don't pollute upstream)
• Ability to propose selected issues upstream
• Support for multiple workers across multiple clones of the same repo

Proposed Solutions

Idea 1: Fork-Aware Hooks + Two-File System

Structure:

# Upstream repo
.beads/
  beads.jsonl          # Canonical frontier (committed)
  .gitignore           # Ignores local.jsonl

# Contributor's fork
.beads/
  beads.jsonl          # Synced from upstream (read-only)
  local.jsonl          # Contributor planning (committed to fork)
  beads.db             # Hydrated from both

Detection: Check for upstream remote to distinguish fork from canonical repo

Workflow:

# In fork
$ bd add "Experiment"        # → local.jsonl (committed to fork)
$ bd sync                    # → Pulls upstream's beads.jsonl
$ bd show                    # → Shows both
$ bd propose bd-a3f8e9       # → Moves issue to beads.jsonl for PR

Pros:

• Git ledger preserved (local.jsonl committed to fork)
• Multi-clone sync works
• Upstream .gitignore prevents pollution

Cons:

• Fork detection doesn't help teams using branches (most common workflow)
• Two files to manage
• Requires discipline to use bd propose

Idea 2: Ownership Metadata + Smart PR Filtering

Structure:

{"id":"bd-123","owner":"upstream","title":"Canonical issue",...}
{"id":"bd-456","owner":"stevey","title":"My planning",...}

Workflow:

$ bd add "Experiment"        # → Creates with owner="stevey"
$ bd propose bd-456          # → Changes owner to "upstream"
$ bd clean-pr                # → Filters commit to only upstream-owned issues
$ git push                   # → PR contains only proposed issues

Pros:

• Single file (simpler)
• Works with any git workflow (branch, fork, etc)
• Git ledger fully preserved

Cons:

• Requires discipline to run bd clean-pr
• Clean commit is awkward (temporarily removing data)
• Merge conflicts if upstream and contributor both modify beads.jsonl

Idea 3: Branch-Scoped Databases

Track which issues belong to which branch, filter at PR time.

Implementation: Similar to #2 but uses labels/metadata to track branch instead of owner.

Challenge: Complex with multiple feature branches, requires tracking branch scope.

Idea 4: Separate Planning Repo (Most Isolated)

Structure:

# Main project repos (many)
~/projects/beads/.beads/beads.jsonl
~/projects/foo/.beads/beads.jsonl

# Single planning repo (one)
~/.beads-planning/.beads/beads.jsonl

# Configuration links them
~/projects/beads/.beads/config.toml:
  planning_repo = "~/.beads-planning"

Workflow:

$ cd ~/projects/beads
$ bd add "My idea"           # → Commits to ~/.beads-planning/
$ bd show                    # → Shows beads canonical + my planning
$ bd propose bd-a3f8e9       # → Exports to beads repo for PR

Pros:

• Complete isolation (separate git histories, zero PR pollution risk)
• Git ledger fully preserved (both repos tracked)
• Multi-clone works perfectly (clone both repos)
• No special filtering/detection needed
• Scales better: One planning repo for all projects

Cons:

• Two repos to manage
• Less obvious for new users (where's my planning?)

Analysis: Fork vs Clone vs Branch

Clone: Local copy of a repo (git clone <url>)

• origin remote points to source
• Push directly to origin (if you have write access)

Fork: Server-side copy on GitHub

• For contributors without write access
• origin → your fork, upstream → original repo
• Push to fork, then PR from fork → upstream

Branch: Feature branches in same repo

• Most common for teams with write access
• Push to same repo, PR from branch → main

Key insight: Branches are universal, forks are only for external contributors. Most teams work on branches in a shared repo.

Current Thinking: Idea 4 is Cleanest

After analysis, separate planning repo (#4) is likely the best solution because:

1. Only solution that truly prevents PR pollution (separate git histories)
2. Git ledger fully preserved (both repos tracked)
3. Multi-clone works perfectly (just clone both)
4. No complex filtering/detection needed (simple config)
5. Better scaling: One planning repo across all projects you contribute to

The "managing two repos" concern is actually an advantage: your planning is centralized and project-agnostic.

Open Questions

About the Workflow

1. Where does PR pollution actually happen?

   • Scenario A: Feature branch → upstream/main includes all beads changes from that branch?
   • Scenario B: Something else?

2. Multi-clone usage pattern:

   • Multiple clones on different machines?
   • All push/pull to same remote?
   • Workers coordinate via git sync?
   • PRs created from feature branches?

About Implementation

1. Proposed issue IDs: When moving issue from planning → canonical, keep same ID? (Hash-based IDs are globally unique)

2. Upstream acceptance sync: If upstream accepts/modifies a proposal, how to sync back to contributor?

   • bd sync detects accepted proposals
   • Moves from planning repo to project's canonical beads.jsonl

3. Multiple projects: One planning repo for all projects you contribute to, or one per project?

4. Backwards compatibility: Single-user projects unchanged (single beads.jsonl)

5. Discovery: How do users discover this feature? Auto-detect and prompt?

Next Steps

Need to clarify:

1. User's actual multi-clone workflow (understand the real use case)
2. Where exactly PR pollution occurs (branch vs fork workflow)
3. Which solution best fits the "git ledger + multi-clone" requirements
4. Whether centralized planning repo (#4) or per-project isolation (#1/#2) is preferred

Design Principles to Preserve

From the conversation, these are non-negotiable:

• Git as ledger: Everything must be git-tracked for forensics and AI repair
• Moving frontier: Focus on active work, aggressively prune completed work
• Multi-clone sync: Workers across clones must coordinate via git
• Small databases: Keep beads.jsonl small enough for agents to read (<25k)
• Simple defaults: Don't break single-user workflows
• Explicit over implicit: Clear boundaries between personal and canonical

---

Decision: Separate Repos (Solution #4)

Date: 2025-11-03 (continued discussion)

Why Separate Repos

After consideration, Solution #4 (Separate Planning Repos) is the chosen approach:

Key Rationale

1. Beads as a Separate Channel: Beads is fundamentally a separate communication channel that happens to use git/VCS for persistence, not a git-centric tool. It should work with any VCS (jujutsu, sapling, mercurial, etc.).

2. VCS-Agnostic Design: Solution #1 (fork detection) is too git-centric and wouldn't work with other version control systems. Separate repos work regardless of VCS.

3. Maximum Flexibility: Supports multiple workflows and personas:

   • OSS contributor with personal planning
   • Multi-phase development (different beads DBs for different stages)
   • Multiple personas (architect, implementer, reviewer)
   • Team vs personal planning separation

4. Zero PR Pollution Risk: Completely separate git histories guarantee no accidental pollution of upstream projects.

5. Proven Pain Point: Experience shows that accidental bulk commits (100k issues) can be catastrophic and traumatic to recover from. Complete isolation is worth the complexity.

Core Architecture Principles

1. Multi-Repo Support (N ≥ 1)

Configuration should support N repos, including N=1 for backward compatibility:

When N=1 (default), this is the current single-repo workflow - no changes needed. When N≥2, multiple repos are hydrated together.

# .beads/config.toml

# Default mode: single repo (backwards compatible)
mode = "single"

# Multi-repo mode
[repos]
  # Primary repo: where canonical issues live
  primary = "."

  # Additional repos to hydrate into the database
  additional = [
    "~/.beads-planning",           # Personal planning across all projects
    "~/.beads-work/phase1",        # Architecting phase
    "~/.beads-work/phase2",        # Implementation phase
    "~/team-shared/.beads",        # Shared team planning
  ]

# Routing: where do new issues go?
[routing]
  mode = "auto"  # auto | explicit
  default = "~/.beads-planning"    # Default for `bd add`

  # Auto-detection: based on user permissions
  [routing.auto]
    maintainer = "."                   # If maintainer, use primary
    contributor = "~/.beads-planning"  # Otherwise use planning repo

2. Hydration Model

On bd show, bd list, etc., the database hydrates from multiple sources:

beads.db ← [
  ./.beads/beads.jsonl           (primary, read-write if maintainer)
  ~/.beads-planning/beads.jsonl   (personal, read-write)
  ~/team-shared/.beads/beads.jsonl (shared, read-write if team member)
]

Metadata tracking:

{
  "id": "bd-a3f8e9",
  "title": "Add dark mode",
  "source_repo": "~/.beads-planning",  # Which repo owns this issue
  "visibility": "local",               # local | proposed | canonical
  ...
}

3. Visibility States

Issues can be in different states of visibility:

• local: Personal planning, only in planning repo
• proposed: Exported for upstream consideration (staged for PR)
• canonical: In the primary repo (upstream accepted it)

4. VCS-Agnostic Operations

Beads should not assume git. Core operations:

• Sync: bd sync should work with git, jj, hg, sl, etc.
• Ledger: Each repo uses whatever VCS it's under (or none)
• Transport: Issues move between repos via export/import, not git-specific operations

Workflow Examples

Use Case 1: OSS Contributor

# One-time setup
$ mkdir ~/.beads-planning
$ cd ~/.beads-planning
$ git init && bd init

# Contributing to upstream project
$ cd ~/projects/some-oss-project
$ bd config --add-repo ~/.beads-planning --routing contributor

# Work
$ bd add "Explore dark mode implementation"
# → Goes to ~/.beads-planning/beads.jsonl
# → Commits to planning repo (git tracked, forensic trail)

$ bd show
# → Shows upstream's canonical issues (read-only)
# → Shows my planning issues (read-write)

$ bd work bd-a3f8e9
$ bd status bd-a3f8e9 in-progress

# Ready to propose
$ bd propose bd-a3f8e9 --target upstream
# → Exports issue from planning repo
# → Creates issue in ./beads/beads.jsonl (staged for PR)
# → Marks as visibility="proposed" in planning repo

$ git add .beads/beads.jsonl
$ git commit -m "Propose: Add dark mode"
$ git push origin feature-branch
# → PR contains only the proposed issue, not all my planning

Use Case 2: Multi-Phase Development

# Setup phases
$ mkdir -p ~/.beads-work/{architecture,implementation,testing}
$ for dir in ~/.beads-work/*; do (cd $dir && git init && bd init); done

# Configure project
$ cd ~/my-big-project
$ bd config --add-repo ~/.beads-work/architecture
$ bd config --add-repo ~/.beads-work/implementation
$ bd config --add-repo ~/.beads-work/testing

# Architecture phase
$ bd add "Design authentication system" --repo ~/.beads-work/architecture
$ bd show --repo ~/.beads-work/architecture
# → Only architecture issues

# Implementation phase (later)
$ bd add "Implement JWT validation" --repo ~/.beads-work/implementation

# View all phases
$ bd show
# → Shows all issues from all configured repos

Use Case 3: Multiple Contributors on Same Project

# Team member Alice (maintainer)
$ cd ~/project
$ bd add "Fix bug in parser"
# → Goes to ./beads/beads.jsonl (she's maintainer)
# → Commits to project repo

# Team member Bob (contributor)
$ cd ~/project
$ bd add "Explore performance optimization"
# → Goes to ~/.beads-planning/beads.jsonl (he's contributor)
# → Does NOT pollute project repo

$ bd show
# → Sees Alice's canonical issue
# → Sees his own planning

$ bd propose bd-xyz
# → Proposes to Alice's canonical repo

Implementation Outline

Phase 1: Core Multi-Repo Support

Commands:

bd config --add-repo <path>         # Add a repo to hydration
bd config --remove-repo <path>      # Remove a repo
bd config --list-repos              # Show all configured repos
bd config --routing <mode>          # Set routing: single|auto|explicit

Config schema:

[repos]
primary = "."
additional = ["path1", "path2", ...]

[routing]
default = "path"  # Where `bd add` goes by default
mode = "auto"     # auto | explicit

Database changes:

• Add source_repo field to issues
• Hydration layer reads from multiple JSONLs
• Writes go to correct JSONL based on source_repo

Phase 2: Proposal Flow

Commands:

bd propose <issue-id> [--target <repo>]   # Move issue to target repo
bd withdraw <issue-id>                    # Un-propose (move back)
bd accept <issue-id>                      # Maintainer accepts proposal

States:

• visibility: local → Personal planning
• visibility: proposed → Staged for PR
• visibility: canonical → Accepted by upstream

Phase 3: Routing Rules

Auto-detection:

• Detect if user is maintainer (git config, permissions)
• Auto-route to primary vs planning repo

Config-based routing (no new schema fields):

[routing]
mode = "auto"  # auto | explicit
default = "~/.beads-planning"  # Fallback for contributors

# Auto-detection rules
[routing.auto]
maintainer = "."  # If user is maintainer, use primary repo
contributor = "~/.beads-planning"  # Otherwise use planning repo

Explicit routing via CLI flag:

# Override auto-detection for specific issues
bd add "Design system" --repo ~/.beads-work/architecture

Discovered issue inheritance:

• Issues with parent_id automatically inherit parent's source_repo
• Keeps related work co-located

Phase 4: VCS-Agnostic Sync

Sync operations:

• Detect VCS type per repo (git, jj, hg, sl)
• Use appropriate sync commands
• Fall back to manual sync if no VCS

Example:

$ bd sync
# Auto-detects:
# - . is git → runs git pull
# - ~/.beads-planning is jj → runs jj git fetch && jj rebase
# - ~/other is hg → runs hg pull && hg update

Migration Path

Existing Users (Single Repo)

No changes required. Current workflow continues to work:

$ bd add "Task"
# → .beads/beads.jsonl (as before)

Opting Into Multi-Repo

# Create planning repo
$ mkdir ~/.beads-planning && cd ~/.beads-planning
$ git init && bd init

# Link to project
$ cd ~/my-project
$ bd config --add-repo ~/.beads-planning
$ bd config --routing auto  # Auto-detect maintainer vs contributor

# Optionally migrate existing issues
$ bd migrate --move-to ~/.beads-planning --filter "author=me"

Teams Adopting Beads

# Maintainer sets up project
$ cd ~/team-project
$ bd init
$ git add .beads/ && git commit -m "Initialize beads"

# Contributors clone and configure
$ git clone team-project
$ cd team-project
$ mkdir ~/.beads-planning && cd ~/.beads-planning
$ git init && bd init
$ cd ~/team-project
$ bd config --add-repo ~/.beads-planning --routing contributor

Design Decisions (Resolved)

1. Namespace Collisions: Option B (Global Uniqueness)

Decision: Use globally unique hash-based IDs that include timestamp + random component.

Rationale (from VC feedback):

• Option C (allow collisions) breaks dependency references: bd dep add bd-a3f8e9 bd-b7c2d1 becomes ambiguous
• Need to support cross-repo dependencies without repo-scoped namespacing
• Hash should be: hash(title + description + timestamp_ms + random_4bytes)
• Collision probability: ~1 in 10^12 (acceptable)

2. Cross-Repo Dependencies: Yes, Fully Supported

Decision: Dependencies work transparently across all repos.

Implementation:

• Hydrated database contains all issues from all repos
• Dependencies stored by ID only (no repo qualifier needed)
• bd ready checks dependency graph across all repos
• Writes route back to correct JSONL via source_repo metadata

3. Routing Mechanism: Config-Based, No Schema Changes

Decision: Use config-based routing + explicit --repo flag. No new schema fields.

Rationale:

• IssueType already exists and is used semantically (bug, feature, task, epic, chore)
• Labels are used semantically by VC (discovered:blocker, no-auto-claim)
• Routing is a storage concern, not issue metadata
• Simpler: auto-detect maintainer vs contributor from config
• Discovered issues inherit parent's source_repo automatically

4. Performance: Smart Caching with File Mtime Tracking

Decision: SQLite DB is the cache, JSONLs are source of truth.

Implementation:

type MultiRepoStorage struct {
    repos      []RepoConfig
    db         *sql.DB
    repoMtimes map[string]time.Time  // Track file modification times
}

func (s *MultiRepoStorage) GetReadyWork(ctx) ([]Issue, error) {
    // Fast path: check if ANY JSONL changed
    needSync := false
    for repo, jsonlPath := range s.jsonlPaths() {
        currentMtime := stat(jsonlPath).ModTime()
        if currentMtime.After(s.repoMtimes[repo]) {
            needSync = true
            s.repoMtimes[repo] = currentMtime
        }
    }

    // Only re-hydrate if something changed
    if needSync {
        s.rehydrate()  // Expensive but rare
    }

    // Query is fast (in-memory SQLite)
    return s.db.Query("SELECT * FROM issues WHERE ...")
}

Rationale: VC's polling loop (every 5-10 seconds) requires sub-second queries. File stat is microseconds, re-parsing only when needed.

5. Visibility Field: Optional, Backward Compatible

Decision: Add visibility as optional field, defaults to "canonical" if missing.

Schema:

type Issue struct {
    // ... existing fields ...
    Visibility *string `json:"visibility,omitempty"`  // nil = canonical
}

States:

• local: Personal planning only
• proposed: Staged for upstream PR
• canonical: Accepted by upstream (or default for existing issues)

Orthogonality: Visibility and Status are independent:

• status: in_progress, visibility: local → Working on personal planning
• status: open, visibility: proposed → Proposed to upstream, awaiting review

6. Library API Stability: Transparent Hydration

Decision: Hybrid approach - transparent by default, explicit opt-in available.

Backward Compatible:

// Existing code keeps working - reads config.toml automatically
store, err := beadsLib.NewSQLiteStorage(".beads/vc.db")

Explicit Override:

// Library consumers can override config
cfg := beadsLib.Config{
    Primary: ".beads/vc.db",
    Additional: []string{"~/.beads-planning"},
}
store, err := beadsLib.NewStorageWithConfig(cfg)

7. ACID Guarantees: Per-Repo File Locking

Decision: Use file-based locks per JSONL, atomic within single repo.

Implementation:

func (s *Storage) UpdateIssue(issue Issue) error {
    sourceRepo := issue.SourceRepo

    // Lock that repo's JSONL
    lock := flock(sourceRepo + "/beads.jsonl.lock")
    defer lock.Unlock()

    // Read-modify-write
    issues := s.readJSONL(sourceRepo)
    issues.Update(issue)
    s.writeJSONL(sourceRepo, issues)

    // Update in-memory DB
    s.db.Update(issue)
}

Limitation: Cross-repo transactions are NOT atomic (acceptable, rare use case).

Key Learnings from VC Feedback

The VC project (VibeCoder) provided critical feedback as a real downstream consumer that uses beads as a library. Key insights:

1. Two Consumer Models

Beads has two distinct consumer types:

• CLI users: Use bd commands directly
• Library consumers: Use beadsLib in Go/TypeScript/etc. (like VC)

Multi-repo must work transparently for both.

2. Performance is Critical for Automation

VC's executor polls GetReadyWork() every 5-10 seconds. Multi-repo hydration must:

• Use smart caching (file mtime tracking)
• Avoid re-parsing JSONLs on every query
• Keep queries sub-second (ideally <100ms)

3. Special Labels Must Work Across Repos

VC uses semantic labels that must work regardless of repo:

• discovered:blocker - Auto-generated blocker issues (priority boost)
• discovered:related - Auto-generated related work
• no-auto-claim - Prevent executor from claiming
• baseline-failure - Self-healing baseline failures

These are semantic labels, not routing labels. Don't overload labels for routing.

4. Discovered Issues Routing

When VC's analysis phase auto-creates issues with discovered:blocker label, they should:

• Inherit parent's source_repo automatically
• Stay co-located with related work
• Not require manual routing decisions

5. Library API Stability is Non-Negotiable

VC's code uses beadsLib.NewSQLiteStorage(). Must not break. Solution:

• Read .beads/config.toml automatically (transparent)
• Provide NewStorageWithConfig() for explicit override
• Hydration happens at storage layer, invisible to library consumers

Remaining Open Questions

1. Sync semantics: When upstream accepts a proposed issue and modifies it, how to sync back?

   • Option A: Mark as "accepted" in planning repo, keep both copies
   • Option B: Delete from planning repo (it's now canonical)
   • Option C: Keep in planning repo but mark as read-only mirror

2. Discovery: How do users learn about this feature?

   • Auto-prompt when detecting fork/contributor status?
   • Docs + examples?
   • bd init --contributor wizard?

3. Metadata fields: Should source_repo be exposed in JSON export, or keep it internal to storage layer?

4. Proposed issue lifecycle: What happens to proposed issues after PR is merged/rejected?

   • Auto-delete from planning repo?
   • Mark as "accepted" or "rejected"?
   • Manual cleanup via bd withdraw?

Success Metrics

How we'll know this works:

1. Zero pollution: No contributor planning issues accidentally merged upstream
2. Multi-clone sync: Workers on different machines see consistent state (via VCS sync)
3. Flexibility: Users can configure for their workflow (personas, phases, etc.)
4. Backwards compat: Existing single-repo users unaffected
5. VCS-agnostic: Works with git, jj, hg, sl, or no VCS

Next Actions

Suggested epics/issues to create (can be done in follow-up session):

1. Epic: Multi-repo hydration layer

   • Design schema for source_repo metadata
   • Implement config parsing for repos.additional
   • Build hydration logic (read from N JSONLs)
   • Build write routing (write to correct JSONL)

2. Epic: Proposal workflow

   • Implement bd propose command
   • Implement bd withdraw command
   • Implement bd accept command (maintainer only)
   • Design visibility state machine

3. Epic: Auto-routing

   • Detect maintainer vs contributor status
   • Implement routing rules (label, priority, custom)
   • Make bd add route to correct repo

4. Epic: VCS-agnostic sync

   • Detect VCS type per repo
   • Implement sync adapters (git, jj, hg, sl)
   • Handle mixed-VCS multi-repo configs

5. Epic: Migration and onboarding

   • Write migration guide
   • Implement bd migrate command
   • Create init wizards for common scenarios
   • Update documentation

---

Summary and Next Steps

This document represents the design evolution for multi-repo support in beads, driven by:

1. Original problem (GitHub #207): Contributors' personal planning pollutes upstream PRs
2. Core insight: Beads is a separate communication channel that happens to use VCS
3. VC feedback: Real-world library consumer with specific performance and API stability needs

Final Architecture

Solution #4 (Separate Repos) with these refinements:

• N ≥ 1 repos: Single repo (N=1) is default, multi-repo is opt-in
• VCS-agnostic: Works with git, jj, hg, sapling, or no VCS
• Config-based routing: No schema changes, auto-detect maintainer vs contributor
• Smart caching: File mtime tracking, SQLite DB as cache layer
• Transparent hydration: Library API remains stable, config-driven
• Global namespace: Hash-based IDs with timestamp + random for uniqueness
• Cross-repo dependencies: Fully supported, transparent to users
• Discovered issues: Inherit parent's source_repo automatically

Why This Design Wins

1. Zero PR pollution: Separate git histories = impossible to accidentally merge planning
2. Git ledger preserved: All repos are VCS-tracked, full forensic capability
3. Maximum flexibility: Supports OSS contributors, multi-phase dev, multi-persona workflows
4. Backward compatible: Existing single-repo users unchanged
5. Performance: Sub-second queries even with polling loops
6. Library-friendly: Transparent to downstream consumers like VC

Related Documents

• Original issue: GitHub #207
• VC feedback: ./vc-feedback-on-multi-repo.md
• Implementation tracking: TBD (epics to be created)

Status

Design: ✅ Complete (pending resolution of open questions) Implementation: ⏳ Not started Target: TBD

Last updated: 2025-11-03