johno/beads
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add multi-repo config schema and source_repo metadata

Browse Source 
For bd-307: Multi-repo hydration layer

Changes:
- Add MultiRepoConfig to internal/config
- Add GetMultiRepoConfig() to retrieve repos.primary and repos.additional
- Add source_repo field to Issue type to track ownership
- Prepare for hydration logic that reads from N repos
This commit is contained in:
Steve Yegge
2025-11-04 12:40:59 -08:00
parent d38a312583
commit 7a1447444c
3 changed files with 35 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.beads/beads.jsonl
View File

@@ -3,7 +3,7 @@
{"id":"bd-1ls","content_hash":"af79708476d560abaac3cbe9c537bcf43c2100dcb1d676954ea604486f75d199","title":"Override test","description":"## Overview\n\n[Describe the high-level goal and scope of this epic]\n\n## Success Criteria\n\n- [ ] Criteria 1\n- [ ] Criteria 2\n- [ ] Criteria 3\n\n## Background\n\n[Provide context and motivation]\n\n## Scope\n\n**In Scope:**\n- Item 1\n- Item 2\n\n**Out of Scope:**\n- Item 1\n- Item 2\n","design":"## Architecture\n\n[Describe the overall architecture and approach]\n\n## Components\n\n- Component 1: [description]\n- Component 2: [description]\n\n## Dependencies\n\n[List external dependencies or constraints]\n","acceptance_criteria":"- [ ] All child issues are completed\n- [ ] Integration tests pass\n- [ ] Documentation is updated\n- [ ] Code review completed\n","status":"closed","priority":1,"issue_type":"bug","created_at":"2025-11-03T20:15:10.01471-08:00","updated_at":"2025-11-03T23:47:58.44067-08:00","closed_at":"2025-11-03T23:47:58.440673-08:00","labels":["epic"]}
{"id":"bd-307","content_hash":"7d47aa1c81710f64c00b7d892f7236fe9f79f94bf56fdf89c0ee1aec30533541","title":"Multi-repo hydration layer","description":"Build core infrastructure to hydrate database from N repos (N≥1), with smart caching via file mtime tracking and routing writes to correct JSONL based on source_repo metadata.","design":"Components:\n- Config schema for repos.additional\n- source_repo metadata field (which repo owns each issue)\n- Hydration logic (read from N JSONLs)\n- Write routing (write to correct JSONL)\n- Smart caching (file mtime tracking to avoid re-parsing)\n- SQLite DB as cache layer","acceptance_criteria":"1. Can configure N repos via config.toml\n2. Database hydrates from all configured repos\n3. Writes route to correct JSONL via source_repo\n4. File mtime caching prevents re-parsing unchanged JSONLs\n5. Query performance \u003c100ms even with 5-10 polling interval\n6. N=1 (single repo) works unchanged","status":"in_progress","priority":1,"issue_type":"epic","created_at":"2025-11-04T11:21:30.655765-08:00","updated_at":"2025-11-04T11:57:32.119534-08:00","dependencies":[{"issue_id":"bd-307","depends_on_id":"bd-4ms","type":"parent-child","created_at":"2025-11-04T11:22:21.823652-08:00","created_by":"daemon"}]}
{"id":"bd-35c7","content_hash":"28e00b560e08ecbf061e998836f8a1dd11392680b273589341c13e6b267df37c","title":"Add label-based filtering to bd ready command","description":"Allow filtering ready work by labels to help organize work by sprint, week, or category.\n\nExample usage:\n bd ready --label week1-2\n bd ready --label frontend,high-priority\n\nThis helps teams organize work into batches and makes it easier for agents to focus on specific categories of work.\n\nImplementation notes:\n- Add --label flag to ready command\n- Support comma-separated labels (AND logic)\n- Should work with existing ready work logic (unblocked issues)","status":"closed","priority":1,"issue_type":"feature","created_at":"2025-11-03T18:10:18.976536-08:00","updated_at":"2025-11-03T23:47:58.441745-08:00","closed_at":"2025-11-03T23:47:58.441748-08:00"}
{"id":"bd-3xq","content_hash":"fc2eca0c4dba35b21ee7ee7e7b20a9b42282cde6fbd71ff42f2d20e0046931e9","title":"Import should not fail on missing parent issues","description":"Edge case discovered during bd-4ms work: auto-import fails completely when JSONL references deleted parent issues.\n\n## The Problem\n\nWhen issues are deleted via bd-delete for hygiene (old/obsolete issues), they remain in git history but not in the database. If the JSONL file contains child issues whose parents were deleted, auto-import fails:\n\n```\nAuto-import failed: error creating issues: parent issue bd-1f4086c5 does not exist\n```\n\nThis blocks ALL imports, even though bd-1f4086c5 exists in git history (findable via commit log).\n\n## Current Behavior\n\nSchema validation is too strict:\n- Hierarchical ID validation requires parent to exist in database\n- Foreign key constraints prevent orphaned child issues\n- Import fails atomically - one bad issue blocks everything\n\nResult: Database stuck at 296 issues, JSONL has more recent data including bd-a101, but can't sync.\n\n## Expected Behavior\n\nBeads should roll forward gracefully:\n1. Import should continue even if parent is missing\n2. Options for handling orphaned children:\n - Skip orphaned issues with warning (log them)\n - Import orphaned issues anyway (relax parent validation)\n - Convert hierarchical ID to top-level (bd-abc.1 → bd-abc-1)\n - Track missing parents and prompt user to restore or clean up\n\n## Root Cause\n\nSchema is trying too hard to be the boss. Valid use cases:\n- Deleting old issues for hygiene (totally OK)\n- Git history preserves deleted issues (forensics still work)\n- JSONL evolution across branches/clones may have different deletion states\n\n## Impact\n\n- Blocks database sync completely\n- Prevents multi-repo work (can't test with current database)\n- Makes bd-delete risky (can break future imports)\n\n## Design Question\n\nHow strict should parent validation be?\n- **Too strict** (current): Fails entire import on missing parent\n- **Too loose**: Allows orphaned children, breaks dependency tree integrity \n- **Just right**: ??? (need to decide)\n\n## Related Work\n\n- bd-a101: Separate branch workflow (blocked by this issue)\n- bd-4ms: Multi-repo support (discovered this while syncing databases)\n- bd-delete command: Makes this scenario possible","status":"open","priority":0,"issue_type":"bug","created_at":"2025-11-04T12:04:01.045251-08:00","updated_at":"2025-11-04T12:04:01.045251-08:00"}
{"id":"bd-3xq","content_hash":"fc2eca0c4dba35b21ee7ee7e7b20a9b42282cde6fbd71ff42f2d20e0046931e9","title":"Import should not fail on missing parent issues","description":"Edge case discovered during bd-4ms work: auto-import fails completely when JSONL references deleted parent issues.\n\n## The Problem\n\nWhen issues are deleted via bd-delete for hygiene (old/obsolete issues), they remain in git history but not in the database. If the JSONL file contains child issues whose parents were deleted, auto-import fails:\n\n```\nAuto-import failed: error creating issues: parent issue bd-1f4086c5 does not exist\n```\n\nThis blocks ALL imports, even though bd-1f4086c5 exists in git history (findable via commit log).\n\n## Current Behavior\n\nSchema validation is too strict:\n- Hierarchical ID validation requires parent to exist in database\n- Foreign key constraints prevent orphaned child issues\n- Import fails atomically - one bad issue blocks everything\n\nResult: Database stuck at 296 issues, JSONL has more recent data including bd-a101, but can't sync.\n\n## Expected Behavior\n\nBeads should roll forward gracefully:\n1. Import should continue even if parent is missing\n2. Options for handling orphaned children:\n - Skip orphaned issues with warning (log them)\n - Import orphaned issues anyway (relax parent validation)\n - Convert hierarchical ID to top-level (bd-abc.1 → bd-abc-1)\n - Track missing parents and prompt user to restore or clean up\n\n## Root Cause\n\nSchema is trying too hard to be the boss. Valid use cases:\n- Deleting old issues for hygiene (totally OK)\n- Git history preserves deleted issues (forensics still work)\n- JSONL evolution across branches/clones may have different deletion states\n\n## Impact\n\n- Blocks database sync completely\n- Prevents multi-repo work (can't test with current database)\n- Makes bd-delete risky (can break future imports)\n\n## Design Question\n\nHow strict should parent validation be?\n- **Too strict** (current): Fails entire import on missing parent\n- **Too loose**: Allows orphaned children, breaks dependency tree integrity \n- **Just right**: ??? (need to decide)\n\n## Related Work\n\n- bd-a101: Separate branch workflow (blocked by this issue)\n- bd-4ms: Multi-repo support (discovered this while syncing databases)\n- bd-delete command: Makes this scenario possible","status":"closed","priority":0,"issue_type":"bug","created_at":"2025-11-04T12:04:01.045251-08:00","updated_at":"2025-11-04T12:39:59.091466-08:00","closed_at":"2025-11-04T12:39:59.091466-08:00"}
{"id":"bd-4ms","content_hash":"557c5d154042ccedeb1442f265dd2b58087469a5b92243ad03fcd868285ecf93","title":"Multi-repo contributor workflow support","description":"Implement separate repository support for OSS contributors to prevent PR pollution while maintaining git ledger and multi-clone sync. Based on contributor-workflow-analysis.md Solution #4.","design":"See docs/contributor-workflow-analysis.md for complete design rationale.\n\nKey architecture:\n- N ≥ 1 repos (single repo default, multi-repo opt-in)\n- VCS-agnostic (git, jj, hg, sapling)\n- Config-based routing (no schema changes)\n- Smart caching (file mtime tracking, SQLite cache)\n- Global namespace (hash-based IDs)\n- Cross-repo dependencies supported","acceptance_criteria":"1. Zero PR pollution - contributor planning never reaches upstream\n2. Git ledger preserved - all repos VCS-tracked\n3. Backward compatible - existing single-repo workflows unchanged\n4. Sub-second query performance with polling\n5. Library API stable - transparent to downstream consumers like VC","status":"in_progress","priority":1,"issue_type":"epic","created_at":"2025-11-04T11:21:19.515776-08:00","updated_at":"2025-11-04T11:57:32.124262-08:00"}
{"id":"bd-4ry","content_hash":"d71a38227ebcb455452f96d07ef100d9c82698aa36c2d4eef5842ca4f1a68fa9","title":"Clarify JSONL size bounds with multi-repo","description":"The contributor-workflow-analysis.md states (line 226): 'Keep beads.jsonl small enough for agents to read (\u003c25k)'\n\nWith multi-repo hydration, it's unclear whether this bound applies to:\n- Each individual JSONL file (likely intention)\n- The total hydrated size across all repos (unclear)\n- Both (most conservative)\n\nClarification needed because:\n- VC monitors .beads/issues.jsonl size to stay under limit\n- With multi-repo, VC needs to know if each additional repo also has 25k limit\n- Agents reading hydrated data need to know total size bounds\n- Performance characteristics depend on total vs per-repo limits\n\nExample scenario:\n- Primary repo: 20k JSONL\n- Planning repo: 15k JSONL\n- Total hydrated: 35k\nIs this acceptable or does it violate the \u003c25k principle?","acceptance_criteria":"- Documentation explicitly states size bound applies per-repo or total\n- Rationale explained (why that bound matters)\n- Guidance for monitoring size with multi-repo\n- If total bound exists, formula provided (e.g., sum of all repos \u003c25k)","status":"open","priority":2,"issue_type":"task","created_at":"2025-11-03T20:24:50.042748-08:00","updated_at":"2025-11-03T20:24:50.042748-08:00"}
{"id":"bd-5c4","content_hash":"d3443a078da2a2f77c3af572c6f43e614b3a62ef0fb87f902529085652df4317","title":"VCS-agnostic sync support","description":"Make bd sync work with multiple VCS types (git, jujutsu, mercurial, sapling) by detecting VCS per repo and using appropriate sync commands, supporting mixed-VCS multi-repo configs.","design":"Features:\n- Detect VCS type per repo (.git, .jj, .hg, .sl)\n- Sync adapters for each VCS type\n- bd sync runs appropriate commands per repo\n- Handle mixed-VCS configs (git + jj + hg)\n- Fallback to manual sync if no VCS detected\n\nExample:\nbd sync auto-detects:\n- . is git → git pull\n- ~/.beads-planning is jj → jj git fetch \u0026\u0026 jj rebase\n- ~/other is hg → hg pull \u0026\u0026 hg update","acceptance_criteria":"1. Detect VCS type per configured repo\n2. bd sync works with git, jj, hg, sapling\n3. Mixed-VCS configs supported\n4. Graceful fallback if no VCS detected\n5. Error messages guide user on manual sync\n6. Pure git workflows unchanged","status":"open","priority":2,"issue_type":"epic","created_at":"2025-11-04T11:22:00.837527-08:00","updated_at":"2025-11-04T11:22:00.837527-08:00","dependencies":[{"issue_id":"bd-5c4","depends_on_id":"bd-4ms","type":"parent-child","created_at":"2025-11-04T11:22:21.817849-08:00","created_by":"daemon"}]}

33
internal/config/config.go
View File

@@ -167,3 +167,36 @@ func AllSettings() map[string]interface{} {
}
return v.AllSettings()
}
// GetStringSlice retrieves a string slice configuration value
func GetStringSlice(key string) []string {
if v == nil {
return []string{}
}
return v.GetStringSlice(key)
}
// MultiRepoConfig contains configuration for multi-repo support
type MultiRepoConfig struct {
Primary string // Primary repo path (where canonical issues live)
Additional []string // Additional repos to hydrate from
}
// GetMultiRepoConfig retrieves multi-repo configuration
// Returns nil if multi-repo is not configured (single-repo mode)
func GetMultiRepoConfig() *MultiRepoConfig {
if v == nil {
return nil
}
// Check if repos.primary is set (indicates multi-repo mode)
primary := v.GetString("repos.primary")
if primary == "" {
return nil // Single-repo mode
}
return &MultiRepoConfig{
Primary: primary,
Additional: v.GetStringSlice("repos.additional"),
}
}

1
internal/types/types.go
View File

@@ -29,6 +29,7 @@ type Issue struct {
CompactedAt *time.Time `json:"compacted_at,omitempty"`
CompactedAtCommit *string `json:"compacted_at_commit,omitempty"` // Git commit hash when compacted
OriginalSize int `json:"original_size,omitempty"`
SourceRepo string `json:"source_repo,omitempty"` // Which repo owns this issue (multi-repo support)
Labels []string `json:"labels,omitempty"` // Populated only for export/import
Dependencies []*Dependency `json:"dependencies,omitempty"` // Populated only for export/import
Comments []*Comment `json:"comments,omitempty"` // Populated only for export/import