new design docs
This commit is contained in:
833
docs/contributor-workflow-analysis.md
Normal file
833
docs/contributor-workflow-analysis.md
Normal file
@@ -0,0 +1,833 @@
|
||||
# Beads Contributor Workflow Analysis
|
||||
|
||||
**Date**: 2025-11-03
|
||||
**Context**: Design discussion on how to handle beads issues in PR/OSS contribution workflows
|
||||
|
||||
## The Problem (from #207)
|
||||
|
||||
When contributing to OSS projects with beads installed:
|
||||
- Git hooks automatically commit contributor's personal planning to PRs
|
||||
- Contributor's experimental musings pollute the upstream project's issue tracker
|
||||
- No clear ownership/permission model for external contributors
|
||||
- Difficult to keep beads changes out of commits
|
||||
|
||||
**Core tension**: Beads is great for team planning (shared namespace), but breaks down for OSS contributions (hierarchical gatekeeping).
|
||||
|
||||
## Key Insights from Discussion
|
||||
|
||||
### Beads as "Moving Frontier"
|
||||
|
||||
Beads is not a traditional issue tracker. It captures the **active working set** - the sliding window of issues currently under attention:
|
||||
|
||||
- Work moves fast with AI agents (10x-50x acceleration)
|
||||
- Completed work fades quickly (95% never revisited, should be pruned aggressively)
|
||||
- Future work is mostly blocked (small frontier of ready tasks)
|
||||
- The frontier is bounded by team size (dozens to hundreds of issues, not thousands)
|
||||
|
||||
**Design principle**: Beads should focus on the "what's next" cloud, not long-term planning or historical archive.
|
||||
|
||||
### The Git Ledger is Fundamental
|
||||
|
||||
Beads achieves reliability despite being unreliable (merge conflicts, sync issues, data staleness) through:
|
||||
|
||||
**A. Git is the ledger and immutable backstop for forensics**
|
||||
**B. AI is the ultimate arbiter and problem-solver when things go wrong**
|
||||
|
||||
Any solution that removes the git ledger (e.g., gitignored contributor files) breaks this model entirely.
|
||||
|
||||
### Requirements for Contributors
|
||||
|
||||
Contributors need:
|
||||
- Git-backed persistence (multi-clone sync, forensics, AI repair)
|
||||
- Isolated planning space (don't pollute upstream)
|
||||
- Ability to propose selected issues upstream
|
||||
- Support for multiple workers across multiple clones of the same repo
|
||||
|
||||
## Proposed Solutions
|
||||
|
||||
### Idea 1: Fork-Aware Hooks + Two-File System
|
||||
|
||||
**Structure**:
|
||||
```
|
||||
# Upstream repo
|
||||
.beads/
|
||||
beads.jsonl # Canonical frontier (committed)
|
||||
.gitignore # Ignores local.jsonl
|
||||
|
||||
# Contributor's fork
|
||||
.beads/
|
||||
beads.jsonl # Synced from upstream (read-only)
|
||||
local.jsonl # Contributor planning (committed to fork)
|
||||
beads.db # Hydrated from both
|
||||
```
|
||||
|
||||
**Detection**: Check for `upstream` remote to distinguish fork from canonical repo
|
||||
|
||||
**Workflow**:
|
||||
```bash
|
||||
# In fork
|
||||
$ bd add "Experiment" # → local.jsonl (committed to fork)
|
||||
$ bd sync # → Pulls upstream's beads.jsonl
|
||||
$ bd show # → Shows both
|
||||
$ bd propose bd-a3f8e9 # → Moves issue to beads.jsonl for PR
|
||||
```
|
||||
|
||||
**Pros**:
|
||||
- Git ledger preserved (local.jsonl committed to fork)
|
||||
- Multi-clone sync works
|
||||
- Upstream .gitignore prevents pollution
|
||||
|
||||
**Cons**:
|
||||
- Fork detection doesn't help teams using branches (most common workflow)
|
||||
- Two files to manage
|
||||
- Requires discipline to use `bd propose`
|
||||
|
||||
### Idea 2: Ownership Metadata + Smart PR Filtering
|
||||
|
||||
**Structure**:
|
||||
```jsonl
|
||||
{"id":"bd-123","owner":"upstream","title":"Canonical issue",...}
|
||||
{"id":"bd-456","owner":"stevey","title":"My planning",...}
|
||||
```
|
||||
|
||||
**Workflow**:
|
||||
```bash
|
||||
$ bd add "Experiment" # → Creates with owner="stevey"
|
||||
$ bd propose bd-456 # → Changes owner to "upstream"
|
||||
$ bd clean-pr # → Filters commit to only upstream-owned issues
|
||||
$ git push # → PR contains only proposed issues
|
||||
```
|
||||
|
||||
**Pros**:
|
||||
- Single file (simpler)
|
||||
- Works with any git workflow (branch, fork, etc)
|
||||
- Git ledger fully preserved
|
||||
|
||||
**Cons**:
|
||||
- Requires discipline to run `bd clean-pr`
|
||||
- Clean commit is awkward (temporarily removing data)
|
||||
- Merge conflicts if upstream and contributor both modify beads.jsonl
|
||||
|
||||
### Idea 3: Branch-Scoped Databases
|
||||
|
||||
Track which issues belong to which branch, filter at PR time.
|
||||
|
||||
**Implementation**: Similar to #2 but uses labels/metadata to track branch instead of owner.
|
||||
|
||||
**Challenge**: Complex with multiple feature branches, requires tracking branch scope.
|
||||
|
||||
### Idea 4: Separate Planning Repo (Most Isolated)
|
||||
|
||||
**Structure**:
|
||||
```bash
|
||||
# Main project repos (many)
|
||||
~/projects/beads/.beads/beads.jsonl
|
||||
~/projects/foo/.beads/beads.jsonl
|
||||
|
||||
# Single planning repo (one)
|
||||
~/.beads-planning/.beads/beads.jsonl
|
||||
|
||||
# Configuration links them
|
||||
~/projects/beads/.beads/config.toml:
|
||||
planning_repo = "~/.beads-planning"
|
||||
```
|
||||
|
||||
**Workflow**:
|
||||
```bash
|
||||
$ cd ~/projects/beads
|
||||
$ bd add "My idea" # → Commits to ~/.beads-planning/
|
||||
$ bd show # → Shows beads canonical + my planning
|
||||
$ bd propose bd-a3f8e9 # → Exports to beads repo for PR
|
||||
```
|
||||
|
||||
**Pros**:
|
||||
- Complete isolation (separate git histories, zero PR pollution risk)
|
||||
- Git ledger fully preserved (both repos tracked)
|
||||
- Multi-clone works perfectly (clone both repos)
|
||||
- No special filtering/detection needed
|
||||
- **Scales better**: One planning repo for all projects
|
||||
|
||||
**Cons**:
|
||||
- Two repos to manage
|
||||
- Less obvious for new users (where's my planning?)
|
||||
|
||||
## Analysis: Fork vs Clone vs Branch
|
||||
|
||||
**Clone**: Local copy of a repo (`git clone <url>`)
|
||||
- `origin` remote points to source
|
||||
- Push directly to origin (if you have write access)
|
||||
|
||||
**Fork**: Server-side copy on GitHub
|
||||
- For contributors without write access
|
||||
- `origin` → your fork, `upstream` → original repo
|
||||
- Push to fork, then PR from fork → upstream
|
||||
|
||||
**Branch**: Feature branches in same repo
|
||||
- Most common for teams with write access
|
||||
- Push to same repo, PR from branch → main
|
||||
|
||||
**Key insight**: Branches are universal, forks are only for external contributors. Most teams work on branches in a shared repo.
|
||||
|
||||
## Current Thinking: Idea 4 is Cleanest
|
||||
|
||||
After analysis, **separate planning repo (#4)** is likely the best solution because:
|
||||
|
||||
1. **Only solution that truly prevents PR pollution** (separate git histories)
|
||||
2. **Git ledger fully preserved** (both repos tracked)
|
||||
3. **Multi-clone works perfectly** (just clone both)
|
||||
4. **No complex filtering/detection needed** (simple config)
|
||||
5. **Better scaling**: One planning repo across all projects you contribute to
|
||||
|
||||
The "managing two repos" concern is actually an advantage: your planning is centralized and project-agnostic.
|
||||
|
||||
## Open Questions
|
||||
|
||||
### About the Workflow
|
||||
|
||||
1. **Where does PR pollution actually happen?**
|
||||
- Scenario A: Feature branch → upstream/main includes all beads changes from that branch?
|
||||
- Scenario B: Something else?
|
||||
|
||||
2. **Multi-clone usage pattern**:
|
||||
- Multiple clones on different machines?
|
||||
- All push/pull to same remote?
|
||||
- Workers coordinate via git sync?
|
||||
- PRs created from feature branches?
|
||||
|
||||
### About Implementation
|
||||
|
||||
1. **Proposed issue IDs**: When moving issue from planning → canonical, keep same ID? (Hash-based IDs are globally unique)
|
||||
|
||||
2. **Upstream acceptance sync**: If upstream accepts/modifies a proposal, how to sync back to contributor?
|
||||
- `bd sync` detects accepted proposals
|
||||
- Moves from planning repo to project's canonical beads.jsonl
|
||||
|
||||
3. **Multiple projects**: One planning repo for all projects you contribute to, or one per project?
|
||||
|
||||
4. **Backwards compatibility**: Single-user projects unchanged (single beads.jsonl)
|
||||
|
||||
5. **Discovery**: How do users discover this feature? Auto-detect and prompt?
|
||||
|
||||
## Next Steps
|
||||
|
||||
Need to clarify:
|
||||
1. User's actual multi-clone workflow (understand the real use case)
|
||||
2. Where exactly PR pollution occurs (branch vs fork workflow)
|
||||
3. Which solution best fits the "git ledger + multi-clone" requirements
|
||||
4. Whether centralized planning repo (#4) or per-project isolation (#1/#2) is preferred
|
||||
|
||||
## Design Principles to Preserve
|
||||
|
||||
From the conversation, these are non-negotiable:
|
||||
|
||||
- **Git as ledger**: Everything must be git-tracked for forensics and AI repair
|
||||
- **Moving frontier**: Focus on active work, aggressively prune completed work
|
||||
- **Multi-clone sync**: Workers across clones must coordinate via git
|
||||
- **Small databases**: Keep beads.jsonl small enough for agents to read (<25k)
|
||||
- **Simple defaults**: Don't break single-user workflows
|
||||
- **Explicit over implicit**: Clear boundaries between personal and canonical
|
||||
|
||||
---
|
||||
|
||||
# Decision: Separate Repos (Solution #4)
|
||||
|
||||
**Date**: 2025-11-03 (continued discussion)
|
||||
|
||||
## Why Separate Repos
|
||||
|
||||
After consideration, **Solution #4 (Separate Planning Repos)** is the chosen approach:
|
||||
|
||||
### Key Rationale
|
||||
|
||||
1. **Beads as a Separate Channel**: Beads is fundamentally a separate communication channel that happens to use git/VCS for persistence, not a git-centric tool. It should work with any VCS (jujutsu, sapling, mercurial, etc.).
|
||||
|
||||
2. **VCS-Agnostic Design**: Solution #1 (fork detection) is too git-centric and wouldn't work with other version control systems. Separate repos work regardless of VCS.
|
||||
|
||||
3. **Maximum Flexibility**: Supports multiple workflows and personas:
|
||||
- OSS contributor with personal planning
|
||||
- Multi-phase development (different beads DBs for different stages)
|
||||
- Multiple personas (architect, implementer, reviewer)
|
||||
- Team vs personal planning separation
|
||||
|
||||
4. **Zero PR Pollution Risk**: Completely separate git histories guarantee no accidental pollution of upstream projects.
|
||||
|
||||
5. **Proven Pain Point**: Experience shows that accidental bulk commits (100k issues) can be catastrophic and traumatic to recover from. Complete isolation is worth the complexity.
|
||||
|
||||
## Core Architecture Principles
|
||||
|
||||
### 1. Multi-Repo Support (N ≥ 1)
|
||||
|
||||
**Configuration should support N repos, including N=1 for backward compatibility:**
|
||||
|
||||
When N=1 (default), this is the current single-repo workflow - no changes needed.
|
||||
When N≥2, multiple repos are hydrated together.
|
||||
|
||||
```toml
|
||||
# .beads/config.toml
|
||||
|
||||
# Default mode: single repo (backwards compatible)
|
||||
mode = "single"
|
||||
|
||||
# Multi-repo mode
|
||||
[repos]
|
||||
# Primary repo: where canonical issues live
|
||||
primary = "."
|
||||
|
||||
# Additional repos to hydrate into the database
|
||||
additional = [
|
||||
"~/.beads-planning", # Personal planning across all projects
|
||||
"~/.beads-work/phase1", # Architecting phase
|
||||
"~/.beads-work/phase2", # Implementation phase
|
||||
"~/team-shared/.beads", # Shared team planning
|
||||
]
|
||||
|
||||
# Routing: where do new issues go?
|
||||
[routing]
|
||||
mode = "auto" # auto | explicit
|
||||
default = "~/.beads-planning" # Default for `bd add`
|
||||
|
||||
# Auto-detection: based on user permissions
|
||||
[routing.auto]
|
||||
maintainer = "." # If maintainer, use primary
|
||||
contributor = "~/.beads-planning" # Otherwise use planning repo
|
||||
```
|
||||
|
||||
### 2. Hydration Model
|
||||
|
||||
On `bd show`, `bd list`, etc., the database hydrates from multiple sources:
|
||||
|
||||
```
|
||||
beads.db ← [
|
||||
./.beads/beads.jsonl (primary, read-write if maintainer)
|
||||
~/.beads-planning/beads.jsonl (personal, read-write)
|
||||
~/team-shared/.beads/beads.jsonl (shared, read-write if team member)
|
||||
]
|
||||
```
|
||||
|
||||
**Metadata tracking**:
|
||||
```jsonl
|
||||
{
|
||||
"id": "bd-a3f8e9",
|
||||
"title": "Add dark mode",
|
||||
"source_repo": "~/.beads-planning", # Which repo owns this issue
|
||||
"visibility": "local", # local | proposed | canonical
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Visibility States
|
||||
|
||||
Issues can be in different states of visibility:
|
||||
|
||||
- **local**: Personal planning, only in planning repo
|
||||
- **proposed**: Exported for upstream consideration (staged for PR)
|
||||
- **canonical**: In the primary repo (upstream accepted it)
|
||||
|
||||
### 4. VCS-Agnostic Operations
|
||||
|
||||
Beads should not assume git. Core operations:
|
||||
|
||||
- **Sync**: `bd sync` should work with git, jj, hg, sl, etc.
|
||||
- **Ledger**: Each repo uses whatever VCS it's under (or none)
|
||||
- **Transport**: Issues move between repos via export/import, not git-specific operations
|
||||
|
||||
## Workflow Examples
|
||||
|
||||
### Use Case 1: OSS Contributor
|
||||
|
||||
```bash
|
||||
# One-time setup
|
||||
$ mkdir ~/.beads-planning
|
||||
$ cd ~/.beads-planning
|
||||
$ git init && bd init
|
||||
|
||||
# Contributing to upstream project
|
||||
$ cd ~/projects/some-oss-project
|
||||
$ bd config --add-repo ~/.beads-planning --routing contributor
|
||||
|
||||
# Work
|
||||
$ bd add "Explore dark mode implementation"
|
||||
# → Goes to ~/.beads-planning/beads.jsonl
|
||||
# → Commits to planning repo (git tracked, forensic trail)
|
||||
|
||||
$ bd show
|
||||
# → Shows upstream's canonical issues (read-only)
|
||||
# → Shows my planning issues (read-write)
|
||||
|
||||
$ bd work bd-a3f8e9
|
||||
$ bd status bd-a3f8e9 in-progress
|
||||
|
||||
# Ready to propose
|
||||
$ bd propose bd-a3f8e9 --target upstream
|
||||
# → Exports issue from planning repo
|
||||
# → Creates issue in ./beads/beads.jsonl (staged for PR)
|
||||
# → Marks as visibility="proposed" in planning repo
|
||||
|
||||
$ git add .beads/beads.jsonl
|
||||
$ git commit -m "Propose: Add dark mode"
|
||||
$ git push origin feature-branch
|
||||
# → PR contains only the proposed issue, not all my planning
|
||||
```
|
||||
|
||||
### Use Case 2: Multi-Phase Development
|
||||
|
||||
```bash
|
||||
# Setup phases
|
||||
$ mkdir -p ~/.beads-work/{architecture,implementation,testing}
|
||||
$ for dir in ~/.beads-work/*; do (cd $dir && git init && bd init); done
|
||||
|
||||
# Configure project
|
||||
$ cd ~/my-big-project
|
||||
$ bd config --add-repo ~/.beads-work/architecture
|
||||
$ bd config --add-repo ~/.beads-work/implementation
|
||||
$ bd config --add-repo ~/.beads-work/testing
|
||||
|
||||
# Architecture phase
|
||||
$ bd add "Design authentication system" --repo ~/.beads-work/architecture
|
||||
$ bd show --repo ~/.beads-work/architecture
|
||||
# → Only architecture issues
|
||||
|
||||
# Implementation phase (later)
|
||||
$ bd add "Implement JWT validation" --repo ~/.beads-work/implementation
|
||||
|
||||
# View all phases
|
||||
$ bd show
|
||||
# → Shows all issues from all configured repos
|
||||
```
|
||||
|
||||
### Use Case 3: Multiple Contributors on Same Project
|
||||
|
||||
```bash
|
||||
# Team member Alice (maintainer)
|
||||
$ cd ~/project
|
||||
$ bd add "Fix bug in parser"
|
||||
# → Goes to ./beads/beads.jsonl (she's maintainer)
|
||||
# → Commits to project repo
|
||||
|
||||
# Team member Bob (contributor)
|
||||
$ cd ~/project
|
||||
$ bd add "Explore performance optimization"
|
||||
# → Goes to ~/.beads-planning/beads.jsonl (he's contributor)
|
||||
# → Does NOT pollute project repo
|
||||
|
||||
$ bd show
|
||||
# → Sees Alice's canonical issue
|
||||
# → Sees his own planning
|
||||
|
||||
$ bd propose bd-xyz
|
||||
# → Proposes to Alice's canonical repo
|
||||
```
|
||||
|
||||
## Implementation Outline
|
||||
|
||||
### Phase 1: Core Multi-Repo Support
|
||||
|
||||
**Commands**:
|
||||
```bash
|
||||
bd config --add-repo <path> # Add a repo to hydration
|
||||
bd config --remove-repo <path> # Remove a repo
|
||||
bd config --list-repos # Show all configured repos
|
||||
bd config --routing <mode> # Set routing: single|auto|explicit
|
||||
```
|
||||
|
||||
**Config schema**:
|
||||
```toml
|
||||
[repos]
|
||||
primary = "."
|
||||
additional = ["path1", "path2", ...]
|
||||
|
||||
[routing]
|
||||
default = "path" # Where `bd add` goes by default
|
||||
mode = "auto" # auto | explicit
|
||||
```
|
||||
|
||||
**Database changes**:
|
||||
- Add `source_repo` field to issues
|
||||
- Hydration layer reads from multiple JSONLs
|
||||
- Writes go to correct JSONL based on source_repo
|
||||
|
||||
### Phase 2: Proposal Flow
|
||||
|
||||
**Commands**:
|
||||
```bash
|
||||
bd propose <issue-id> [--target <repo>] # Move issue to target repo
|
||||
bd withdraw <issue-id> # Un-propose (move back)
|
||||
bd accept <issue-id> # Maintainer accepts proposal
|
||||
```
|
||||
|
||||
**States**:
|
||||
- `visibility: local` → Personal planning
|
||||
- `visibility: proposed` → Staged for PR
|
||||
- `visibility: canonical` → Accepted by upstream
|
||||
|
||||
### Phase 3: Routing Rules
|
||||
|
||||
**Auto-detection**:
|
||||
- Detect if user is maintainer (git config, permissions)
|
||||
- Auto-route to primary vs planning repo
|
||||
|
||||
**Config-based routing** (no new schema fields):
|
||||
```toml
|
||||
[routing]
|
||||
mode = "auto" # auto | explicit
|
||||
default = "~/.beads-planning" # Fallback for contributors
|
||||
|
||||
# Auto-detection rules
|
||||
[routing.auto]
|
||||
maintainer = "." # If user is maintainer, use primary repo
|
||||
contributor = "~/.beads-planning" # Otherwise use planning repo
|
||||
```
|
||||
|
||||
**Explicit routing** via CLI flag:
|
||||
```bash
|
||||
# Override auto-detection for specific issues
|
||||
bd add "Design system" --repo ~/.beads-work/architecture
|
||||
```
|
||||
|
||||
**Discovered issue inheritance**:
|
||||
- Issues with parent_id automatically inherit parent's source_repo
|
||||
- Keeps related work co-located
|
||||
|
||||
### Phase 4: VCS-Agnostic Sync
|
||||
|
||||
**Sync operations**:
|
||||
- Detect VCS type per repo (git, jj, hg, sl)
|
||||
- Use appropriate sync commands
|
||||
- Fall back to manual sync if no VCS
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
$ bd sync
|
||||
# Auto-detects:
|
||||
# - . is git → runs git pull
|
||||
# - ~/.beads-planning is jj → runs jj git fetch && jj rebase
|
||||
# - ~/other is hg → runs hg pull && hg update
|
||||
```
|
||||
|
||||
## Migration Path
|
||||
|
||||
### Existing Users (Single Repo)
|
||||
|
||||
No changes required. Current workflow continues to work:
|
||||
|
||||
```bash
|
||||
$ bd add "Task"
|
||||
# → .beads/beads.jsonl (as before)
|
||||
```
|
||||
|
||||
### Opting Into Multi-Repo
|
||||
|
||||
```bash
|
||||
# Create planning repo
|
||||
$ mkdir ~/.beads-planning && cd ~/.beads-planning
|
||||
$ git init && bd init
|
||||
|
||||
# Link to project
|
||||
$ cd ~/my-project
|
||||
$ bd config --add-repo ~/.beads-planning
|
||||
$ bd config --routing auto # Auto-detect maintainer vs contributor
|
||||
|
||||
# Optionally migrate existing issues
|
||||
$ bd migrate --move-to ~/.beads-planning --filter "author=me"
|
||||
```
|
||||
|
||||
### Teams Adopting Beads
|
||||
|
||||
```bash
|
||||
# Maintainer sets up project
|
||||
$ cd ~/team-project
|
||||
$ bd init
|
||||
$ git add .beads/ && git commit -m "Initialize beads"
|
||||
|
||||
# Contributors clone and configure
|
||||
$ git clone team-project
|
||||
$ cd team-project
|
||||
$ mkdir ~/.beads-planning && cd ~/.beads-planning
|
||||
$ git init && bd init
|
||||
$ cd ~/team-project
|
||||
$ bd config --add-repo ~/.beads-planning --routing contributor
|
||||
```
|
||||
|
||||
## Design Decisions (Resolved)
|
||||
|
||||
### 1. Namespace Collisions: **Option B (Global Uniqueness)**
|
||||
|
||||
**Decision**: Use globally unique hash-based IDs that include timestamp + random component.
|
||||
|
||||
**Rationale** (from VC feedback):
|
||||
- Option C (allow collisions) breaks dependency references: `bd dep add bd-a3f8e9 bd-b7c2d1` becomes ambiguous
|
||||
- Need to support cross-repo dependencies without repo-scoped namespacing
|
||||
- Hash should be: `hash(title + description + timestamp_ms + random_4bytes)`
|
||||
- Collision probability: ~1 in 10^12 (acceptable)
|
||||
|
||||
### 2. Cross-Repo Dependencies: **Yes, Fully Supported**
|
||||
|
||||
**Decision**: Dependencies work transparently across all repos.
|
||||
|
||||
**Implementation**:
|
||||
- Hydrated database contains all issues from all repos
|
||||
- Dependencies stored by ID only (no repo qualifier needed)
|
||||
- `bd ready` checks dependency graph across all repos
|
||||
- Writes route back to correct JSONL via `source_repo` metadata
|
||||
|
||||
### 3. Routing Mechanism: **Config-Based, No Schema Changes**
|
||||
|
||||
**Decision**: Use config-based routing + explicit `--repo` flag. No new schema fields.
|
||||
|
||||
**Rationale**:
|
||||
- `IssueType` already exists and is used semantically (bug, feature, task, epic, chore)
|
||||
- Labels are used semantically by VC (`discovered:blocker`, `no-auto-claim`)
|
||||
- Routing is a storage concern, not issue metadata
|
||||
- Simpler: auto-detect maintainer vs contributor from config
|
||||
- Discovered issues inherit parent's `source_repo` automatically
|
||||
|
||||
### 4. Performance: **Smart Caching with File Mtime Tracking**
|
||||
|
||||
**Decision**: SQLite DB is the cache, JSONLs are source of truth.
|
||||
|
||||
**Implementation**:
|
||||
```go
|
||||
type MultiRepoStorage struct {
|
||||
repos []RepoConfig
|
||||
db *sql.DB
|
||||
repoMtimes map[string]time.Time // Track file modification times
|
||||
}
|
||||
|
||||
func (s *MultiRepoStorage) GetReadyWork(ctx) ([]Issue, error) {
|
||||
// Fast path: check if ANY JSONL changed
|
||||
needSync := false
|
||||
for repo, jsonlPath := range s.jsonlPaths() {
|
||||
currentMtime := stat(jsonlPath).ModTime()
|
||||
if currentMtime.After(s.repoMtimes[repo]) {
|
||||
needSync = true
|
||||
s.repoMtimes[repo] = currentMtime
|
||||
}
|
||||
}
|
||||
|
||||
// Only re-hydrate if something changed
|
||||
if needSync {
|
||||
s.rehydrate() // Expensive but rare
|
||||
}
|
||||
|
||||
// Query is fast (in-memory SQLite)
|
||||
return s.db.Query("SELECT * FROM issues WHERE ...")
|
||||
}
|
||||
```
|
||||
|
||||
**Rationale**: VC's polling loop (every 5-10 seconds) requires sub-second queries. File stat is microseconds, re-parsing only when needed.
|
||||
|
||||
### 5. Visibility Field: **Optional, Backward Compatible**
|
||||
|
||||
**Decision**: Add `visibility` as optional field, defaults to "canonical" if missing.
|
||||
|
||||
**Schema**:
|
||||
```go
|
||||
type Issue struct {
|
||||
// ... existing fields ...
|
||||
Visibility *string `json:"visibility,omitempty"` // nil = canonical
|
||||
}
|
||||
```
|
||||
|
||||
**States**:
|
||||
- `local`: Personal planning only
|
||||
- `proposed`: Staged for upstream PR
|
||||
- `canonical`: Accepted by upstream (or default for existing issues)
|
||||
|
||||
**Orthogonality**: Visibility and Status are independent:
|
||||
- `status: in_progress, visibility: local` → Working on personal planning
|
||||
- `status: open, visibility: proposed` → Proposed to upstream, awaiting review
|
||||
|
||||
### 6. Library API Stability: **Transparent Hydration**
|
||||
|
||||
**Decision**: Hybrid approach - transparent by default, explicit opt-in available.
|
||||
|
||||
**Backward Compatible**:
|
||||
```go
|
||||
// Existing code keeps working - reads config.toml automatically
|
||||
store, err := beadsLib.NewSQLiteStorage(".beads/vc.db")
|
||||
```
|
||||
|
||||
**Explicit Override**:
|
||||
```go
|
||||
// Library consumers can override config
|
||||
cfg := beadsLib.Config{
|
||||
Primary: ".beads/vc.db",
|
||||
Additional: []string{"~/.beads-planning"},
|
||||
}
|
||||
store, err := beadsLib.NewStorageWithConfig(cfg)
|
||||
```
|
||||
|
||||
### 7. ACID Guarantees: **Per-Repo File Locking**
|
||||
|
||||
**Decision**: Use file-based locks per JSONL, atomic within single repo.
|
||||
|
||||
**Implementation**:
|
||||
```go
|
||||
func (s *Storage) UpdateIssue(issue Issue) error {
|
||||
sourceRepo := issue.SourceRepo
|
||||
|
||||
// Lock that repo's JSONL
|
||||
lock := flock(sourceRepo + "/beads.jsonl.lock")
|
||||
defer lock.Unlock()
|
||||
|
||||
// Read-modify-write
|
||||
issues := s.readJSONL(sourceRepo)
|
||||
issues.Update(issue)
|
||||
s.writeJSONL(sourceRepo, issues)
|
||||
|
||||
// Update in-memory DB
|
||||
s.db.Update(issue)
|
||||
}
|
||||
```
|
||||
|
||||
**Limitation**: Cross-repo transactions are NOT atomic (acceptable, rare use case).
|
||||
|
||||
## Key Learnings from VC Feedback
|
||||
|
||||
The VC project (VibeCoder) provided critical feedback as a real downstream consumer that uses beads as a library. Key insights:
|
||||
|
||||
### 1. Two Consumer Models
|
||||
|
||||
Beads has two distinct consumer types:
|
||||
- **CLI users**: Use `bd` commands directly
|
||||
- **Library consumers**: Use `beadsLib` in Go/TypeScript/etc. (like VC)
|
||||
|
||||
Multi-repo must work transparently for both.
|
||||
|
||||
### 2. Performance is Critical for Automation
|
||||
|
||||
VC's executor polls `GetReadyWork()` every 5-10 seconds. Multi-repo hydration must:
|
||||
- Use smart caching (file mtime tracking)
|
||||
- Avoid re-parsing JSONLs on every query
|
||||
- Keep queries sub-second (ideally <100ms)
|
||||
|
||||
### 3. Special Labels Must Work Across Repos
|
||||
|
||||
VC uses semantic labels that must work regardless of repo:
|
||||
- `discovered:blocker` - Auto-generated blocker issues (priority boost)
|
||||
- `discovered:related` - Auto-generated related work
|
||||
- `no-auto-claim` - Prevent executor from claiming
|
||||
- `baseline-failure` - Self-healing baseline failures
|
||||
|
||||
These are **semantic labels**, not routing labels. Don't overload labels for routing.
|
||||
|
||||
### 4. Discovered Issues Routing
|
||||
|
||||
When VC's analysis phase auto-creates issues with `discovered:blocker` label, they should:
|
||||
- Inherit parent's `source_repo` automatically
|
||||
- Stay co-located with related work
|
||||
- Not require manual routing decisions
|
||||
|
||||
### 5. Library API Stability is Non-Negotiable
|
||||
|
||||
VC's code uses `beadsLib.NewSQLiteStorage()`. Must not break. Solution:
|
||||
- Read `.beads/config.toml` automatically (transparent)
|
||||
- Provide `NewStorageWithConfig()` for explicit override
|
||||
- Hydration happens at storage layer, invisible to library consumers
|
||||
|
||||
## Remaining Open Questions
|
||||
|
||||
1. **Sync semantics**: When upstream accepts a proposed issue and modifies it, how to sync back?
|
||||
- Option A: Mark as "accepted" in planning repo, keep both copies
|
||||
- Option B: Delete from planning repo (it's now canonical)
|
||||
- Option C: Keep in planning repo but mark as read-only mirror
|
||||
|
||||
2. **Discovery**: How do users learn about this feature?
|
||||
- Auto-prompt when detecting fork/contributor status?
|
||||
- Docs + examples?
|
||||
- `bd init --contributor` wizard?
|
||||
|
||||
3. **Metadata fields**: Should `source_repo` be exposed in JSON export, or keep it internal to storage layer?
|
||||
|
||||
4. **Proposed issue lifecycle**: What happens to proposed issues after PR is merged/rejected?
|
||||
- Auto-delete from planning repo?
|
||||
- Mark as "accepted" or "rejected"?
|
||||
- Manual cleanup via `bd withdraw`?
|
||||
|
||||
## Success Metrics
|
||||
|
||||
How we'll know this works:
|
||||
|
||||
1. **Zero pollution**: No contributor planning issues accidentally merged upstream
|
||||
2. **Multi-clone sync**: Workers on different machines see consistent state (via VCS sync)
|
||||
3. **Flexibility**: Users can configure for their workflow (personas, phases, etc.)
|
||||
4. **Backwards compat**: Existing single-repo users unaffected
|
||||
5. **VCS-agnostic**: Works with git, jj, hg, sl, or no VCS
|
||||
|
||||
## Next Actions
|
||||
|
||||
Suggested epics/issues to create (can be done in follow-up session):
|
||||
|
||||
1. **Epic: Multi-repo hydration layer**
|
||||
- Design schema for source_repo metadata
|
||||
- Implement config parsing for repos.additional
|
||||
- Build hydration logic (read from N JSONLs)
|
||||
- Build write routing (write to correct JSONL)
|
||||
|
||||
2. **Epic: Proposal workflow**
|
||||
- Implement `bd propose` command
|
||||
- Implement `bd withdraw` command
|
||||
- Implement `bd accept` command (maintainer only)
|
||||
- Design visibility state machine
|
||||
|
||||
3. **Epic: Auto-routing**
|
||||
- Detect maintainer vs contributor status
|
||||
- Implement routing rules (label, priority, custom)
|
||||
- Make `bd add` route to correct repo
|
||||
|
||||
4. **Epic: VCS-agnostic sync**
|
||||
- Detect VCS type per repo
|
||||
- Implement sync adapters (git, jj, hg, sl)
|
||||
- Handle mixed-VCS multi-repo configs
|
||||
|
||||
5. **Epic: Migration and onboarding**
|
||||
- Write migration guide
|
||||
- Implement `bd migrate` command
|
||||
- Create init wizards for common scenarios
|
||||
- Update documentation
|
||||
|
||||
---
|
||||
|
||||
## Summary and Next Steps
|
||||
|
||||
This document represents the design evolution for multi-repo support in beads, driven by:
|
||||
|
||||
1. **Original problem** (GitHub #207): Contributors' personal planning pollutes upstream PRs
|
||||
2. **Core insight**: Beads is a separate communication channel that happens to use VCS
|
||||
3. **VC feedback**: Real-world library consumer with specific performance and API stability needs
|
||||
|
||||
### Final Architecture
|
||||
|
||||
**Solution #4 (Separate Repos)** with these refinements:
|
||||
|
||||
- **N ≥ 1 repos**: Single repo (N=1) is default, multi-repo is opt-in
|
||||
- **VCS-agnostic**: Works with git, jj, hg, sapling, or no VCS
|
||||
- **Config-based routing**: No schema changes, auto-detect maintainer vs contributor
|
||||
- **Smart caching**: File mtime tracking, SQLite DB as cache layer
|
||||
- **Transparent hydration**: Library API remains stable, config-driven
|
||||
- **Global namespace**: Hash-based IDs with timestamp + random for uniqueness
|
||||
- **Cross-repo dependencies**: Fully supported, transparent to users
|
||||
- **Discovered issues**: Inherit parent's source_repo automatically
|
||||
|
||||
### Why This Design Wins
|
||||
|
||||
1. **Zero PR pollution**: Separate git histories = impossible to accidentally merge planning
|
||||
2. **Git ledger preserved**: All repos are VCS-tracked, full forensic capability
|
||||
3. **Maximum flexibility**: Supports OSS contributors, multi-phase dev, multi-persona workflows
|
||||
4. **Backward compatible**: Existing single-repo users unchanged
|
||||
5. **Performance**: Sub-second queries even with polling loops
|
||||
6. **Library-friendly**: Transparent to downstream consumers like VC
|
||||
|
||||
### Related Documents
|
||||
|
||||
- Original issue: GitHub #207
|
||||
- VC feedback: `./vc-feedback-on-multi-repo.md`
|
||||
- Implementation tracking: TBD (epics to be created)
|
||||
|
||||
### Status
|
||||
|
||||
**Design**: ✅ Complete (pending resolution of open questions)
|
||||
**Implementation**: ⏳ Not started
|
||||
**Target**: TBD
|
||||
|
||||
Last updated: 2025-11-03
|
||||
Reference in New Issue
Block a user