Add UnderlyingDB() method for extension database access
Implements database platform layer for extensions like VC to create their own tables in the same SQLite database. Changes: - Add UnderlyingDB() *sql.DB to Storage interface - Implement in SQLiteStorage to expose underlying connection - Add comprehensive test suite (5 tests, -race clean) - Tests cover: basic access, extension tables, concurrency, lifecycle safety, and transaction behavior This allows VC to host its executor_instances and other tables alongside beads core tables with proper FK enforcement. Related issues: bd-57, bd-64, bd-65, bd-66 Amp-Thread-ID: https://ampcode.com/threads/T-a6715beb-fe92-4dee-b931-3c9327124875 Co-authored-by: Amp <amp@ampcode.com>
This commit is contained in:
@@ -58,6 +58,9 @@
|
|||||||
{"id":"bd-61","title":"Phase 3: Migration Path \u0026 Database Schema Alignment","description":"Enable existing .beads/vc.db files to work with Beads library through automated migration.\n\n**Goal:** Provide safe, tested migration path from SQLite implementation to Beads library.\n\n**Key Tasks:**\n1. Run compatibility tests against production databases\n2. Identify schema differences (columns, indexes, constraints)\n3. Document required migrations\n4. Create migration CLI command: 'vc migrate --from sqlite --to beads'\n5. Add dry-run mode for preview\n6. Add backup/restore capability\n7. Implement rollback mechanism\n8. Add auto-detection of schema version on startup\n9. Add auto-migrate with user prompt\n\n**Acceptance Criteria:**\n- Existing databases migrate successfully\n- Data integrity preserved (zero data loss verified via checksums)\n- Rollback works if migration fails\n- Migration tested on real production VC databases\n- Dry-run mode shows exactly what will change\n- Backup created before migration\n- Feature flag: VC_FORCE_SQLITE=true provides escape hatch\n\n**Technical Details:**\n- Compare current SQLite schema with Beads schema\n- Handle version detection (read schema_version or detect from structure)\n- Migration should be idempotent (safe to run multiple times)\n- Backup strategy: Copy .beads/vc.db to .beads/vc.db.backup-\u003ctimestamp\u003e\n- Verify foreign key integrity after migration\n\n**Safety Measures:**\n- Require executor shutdown before migration (check for running executors)\n- Atomic migration (BEGIN IMMEDIATE transaction)\n- Comprehensive pre/post migration validation\n- Clear error messages with recovery instructions\n\n**Dependencies:**\n- Blocked by Phase 2 (need VCStorage implementation)\n\n**Estimated Effort:** 0.5 sprint","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-22T14:04:51.320435-07:00","updated_at":"2025-10-22T14:04:51.320435-07:00","dependencies":[{"issue_id":"bd-61","depends_on_id":"bd-58","type":"parent-child","created_at":"2025-10-22T14:04:51.321526-07:00","created_by":"daemon"},{"issue_id":"bd-61","depends_on_id":"bd-60","type":"blocks","created_at":"2025-10-22T14:04:51.321935-07:00","created_by":"daemon"}]}
|
{"id":"bd-61","title":"Phase 3: Migration Path \u0026 Database Schema Alignment","description":"Enable existing .beads/vc.db files to work with Beads library through automated migration.\n\n**Goal:** Provide safe, tested migration path from SQLite implementation to Beads library.\n\n**Key Tasks:**\n1. Run compatibility tests against production databases\n2. Identify schema differences (columns, indexes, constraints)\n3. Document required migrations\n4. Create migration CLI command: 'vc migrate --from sqlite --to beads'\n5. Add dry-run mode for preview\n6. Add backup/restore capability\n7. Implement rollback mechanism\n8. Add auto-detection of schema version on startup\n9. Add auto-migrate with user prompt\n\n**Acceptance Criteria:**\n- Existing databases migrate successfully\n- Data integrity preserved (zero data loss verified via checksums)\n- Rollback works if migration fails\n- Migration tested on real production VC databases\n- Dry-run mode shows exactly what will change\n- Backup created before migration\n- Feature flag: VC_FORCE_SQLITE=true provides escape hatch\n\n**Technical Details:**\n- Compare current SQLite schema with Beads schema\n- Handle version detection (read schema_version or detect from structure)\n- Migration should be idempotent (safe to run multiple times)\n- Backup strategy: Copy .beads/vc.db to .beads/vc.db.backup-\u003ctimestamp\u003e\n- Verify foreign key integrity after migration\n\n**Safety Measures:**\n- Require executor shutdown before migration (check for running executors)\n- Atomic migration (BEGIN IMMEDIATE transaction)\n- Comprehensive pre/post migration validation\n- Clear error messages with recovery instructions\n\n**Dependencies:**\n- Blocked by Phase 2 (need VCStorage implementation)\n\n**Estimated Effort:** 0.5 sprint","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-22T14:04:51.320435-07:00","updated_at":"2025-10-22T14:04:51.320435-07:00","dependencies":[{"issue_id":"bd-61","depends_on_id":"bd-58","type":"parent-child","created_at":"2025-10-22T14:04:51.321526-07:00","created_by":"daemon"},{"issue_id":"bd-61","depends_on_id":"bd-60","type":"blocks","created_at":"2025-10-22T14:04:51.321935-07:00","created_by":"daemon"}]}
|
||||||
{"id":"bd-62","title":"Phase 4: Gradual Cutover \u0026 Production Rollout","description":"Replace SQLite implementation with Beads library in production and remove legacy code.\n\n**Goal:** Complete transition to Beads library, deprecate and remove custom SQLite implementation.\n\n**Key Tasks:**\n1. Run VC executor with Beads library in CI\n2. Dogfood: Use Beads library for VC's own development\n3. Monitor for regressions and performance issues\n4. Flip feature flag: VC_USE_BEADS_LIBRARY=true by default\n5. Monitor production logs for errors\n6. Collect user feedback\n7. Add deprecation notice to CLAUDE.md\n8. Provide migration guide for users\n9. Remove legacy code: internal/storage/sqlite/sqlite.go (~1500 lines)\n10. Remove migration framework: internal/storage/migrations/\n11. Remove manual transaction management code\n12. Update all documentation\n\n**Acceptance Criteria:**\n- Beads library enabled by default in production\n- Zero production incidents related to migration\n- Performance meets or exceeds SQLite implementation\n- All tests passing with Beads library\n- Legacy SQLite code removed\n- Documentation updated\n- Celebration documented 🎉\n\n**Rollout Strategy:**\n1. Week 1: Enable for CI/testing environments\n2. Week 2: Dogfood on VC development\n3. Week 3: Enable for 50% of production (canary)\n4. Week 4: Enable for 100% of production\n5. Week 5: Remove legacy code\n\n**Monitoring:**\n- Track error rates before/after cutover\n- Monitor database query performance\n- Track issue creation/update latency\n- Monitor executor claim performance\n\n**Rollback Plan:**\n- Keep VC_FORCE_SQLITE=true escape hatch for 2 weeks post-cutover\n- Keep legacy code for 1 sprint after cutover\n- Document rollback procedure\n\n**Success Metrics:**\n- Zero data loss\n- No performance regression (\u003c 5% latency increase acceptable)\n- Reduced maintenance burden (code LOC reduction)\n- Positive developer feedback\n\n**Dependencies:**\n- Blocked by Phase 3 (need migration tooling)\n\n**Estimated Effort:** 1 sprint","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-22T14:05:07.755107-07:00","updated_at":"2025-10-22T14:05:07.755107-07:00","dependencies":[{"issue_id":"bd-62","depends_on_id":"bd-58","type":"parent-child","created_at":"2025-10-22T14:05:07.756023-07:00","created_by":"daemon"},{"issue_id":"bd-62","depends_on_id":"bd-61","type":"blocks","created_at":"2025-10-22T14:05:07.75651-07:00","created_by":"daemon"}]}
|
{"id":"bd-62","title":"Phase 4: Gradual Cutover \u0026 Production Rollout","description":"Replace SQLite implementation with Beads library in production and remove legacy code.\n\n**Goal:** Complete transition to Beads library, deprecate and remove custom SQLite implementation.\n\n**Key Tasks:**\n1. Run VC executor with Beads library in CI\n2. Dogfood: Use Beads library for VC's own development\n3. Monitor for regressions and performance issues\n4. Flip feature flag: VC_USE_BEADS_LIBRARY=true by default\n5. Monitor production logs for errors\n6. Collect user feedback\n7. Add deprecation notice to CLAUDE.md\n8. Provide migration guide for users\n9. Remove legacy code: internal/storage/sqlite/sqlite.go (~1500 lines)\n10. Remove migration framework: internal/storage/migrations/\n11. Remove manual transaction management code\n12. Update all documentation\n\n**Acceptance Criteria:**\n- Beads library enabled by default in production\n- Zero production incidents related to migration\n- Performance meets or exceeds SQLite implementation\n- All tests passing with Beads library\n- Legacy SQLite code removed\n- Documentation updated\n- Celebration documented 🎉\n\n**Rollout Strategy:**\n1. Week 1: Enable for CI/testing environments\n2. Week 2: Dogfood on VC development\n3. Week 3: Enable for 50% of production (canary)\n4. Week 4: Enable for 100% of production\n5. Week 5: Remove legacy code\n\n**Monitoring:**\n- Track error rates before/after cutover\n- Monitor database query performance\n- Track issue creation/update latency\n- Monitor executor claim performance\n\n**Rollback Plan:**\n- Keep VC_FORCE_SQLITE=true escape hatch for 2 weeks post-cutover\n- Keep legacy code for 1 sprint after cutover\n- Document rollback procedure\n\n**Success Metrics:**\n- Zero data loss\n- No performance regression (\u003c 5% latency increase acceptable)\n- Reduced maintenance burden (code LOC reduction)\n- Positive developer feedback\n\n**Dependencies:**\n- Blocked by Phase 3 (need migration tooling)\n\n**Estimated Effort:** 1 sprint","status":"open","priority":2,"issue_type":"task","created_at":"2025-10-22T14:05:07.755107-07:00","updated_at":"2025-10-22T14:05:07.755107-07:00","dependencies":[{"issue_id":"bd-62","depends_on_id":"bd-58","type":"parent-child","created_at":"2025-10-22T14:05:07.756023-07:00","created_by":"daemon"},{"issue_id":"bd-62","depends_on_id":"bd-61","type":"blocks","created_at":"2025-10-22T14:05:07.75651-07:00","created_by":"daemon"}]}
|
||||||
{"id":"bd-63","title":"Example library-created issue","description":"This issue was created programmatically using Beads as a library","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-22T14:34:44.081801-07:00","updated_at":"2025-10-22T14:34:44.084241-07:00","closed_at":"2025-10-22T14:34:44.084241-07:00","labels":["library-usage"],"dependencies":[{"issue_id":"bd-63","depends_on_id":"bd-1","type":"discovered-from","created_at":"2025-10-22T14:34:44.082772-07:00","created_by":"library-example"}],"comments":[{"id":7,"issue_id":"bd-63","author":"library-example","text":"This is a programmatic comment","created_at":"2025-10-22T21:34:44Z"}]}
|
{"id":"bd-63","title":"Example library-created issue","description":"This issue was created programmatically using Beads as a library","status":"closed","priority":2,"issue_type":"task","created_at":"2025-10-22T14:34:44.081801-07:00","updated_at":"2025-10-22T14:34:44.084241-07:00","closed_at":"2025-10-22T14:34:44.084241-07:00","labels":["library-usage"],"dependencies":[{"issue_id":"bd-63","depends_on_id":"bd-1","type":"discovered-from","created_at":"2025-10-22T14:34:44.082772-07:00","created_by":"library-example"}],"comments":[{"id":7,"issue_id":"bd-63","author":"library-example","text":"This is a programmatic comment","created_at":"2025-10-22T21:34:44Z"}]}
|
||||||
|
{"id":"bd-64","title":"Add lifecycle safety docs and tests for UnderlyingDB() method","description":"The new UnderlyingDB() method exposes the raw *sql.DB connection for extensions like VC to create their own tables. While database/sql is concurrency-safe, there are lifecycle and misuse risks that need documentation and testing.\n\n**What needs to be done:**\n\n1. **Enhanced documentation** - Expand UnderlyingDB() comments to warn:\n - Callers MUST NOT call Close() on returned DB\n - Do NOT change pool/driver settings (SetMaxOpenConns, SetConnMaxIdleTime)\n - Do NOT modify SQLite PRAGMAs (WAL mode, journal, etc.)\n - Expect errors after Storage.Close() - use contexts\n - Keep write transactions short to avoid blocking core storage\n\n2. **Add lifecycle tracking** - Implement closed flag:\n - Add atomic.Bool closed field to SQLiteStorage\n - Set flag in Close(), clear in New()\n - Optional: Add IsClosed() bool method\n\n3. **Add safety tests** (run with -race):\n - TestUnderlyingDB_ConcurrentAccess - N goroutines using UnderlyingDB() during normal storage ops\n - TestUnderlyingDB_AfterClose - Verify operations fail cleanly after storage closed\n - TestUnderlyingDB_CreateExtensionTables - Create VC table with FK to issues, verify FK enforcement\n - TestUnderlyingDB_LongTxDoesNotCorrupt - Ensure long read tx doesn't block writes indefinitely\n\n**Why this matters:**\nVC will use this to create tables in the same database. Need to ensure production-ready safety without over-engineering.\n\n**Estimated effort:** S+S+S = M total (1-3h)","design":"Oracle recommends \"simple path\": enhanced docs + minimal guardrails + focused tests. See oracle output for detailed rationale on concurrency safety, lifecycle risks, and when to consider advanced path (wrapping interface).","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-22T17:07:56.812983-07:00","updated_at":"2025-10-22T17:07:56.812983-07:00"}
|
||||||
|
{"id":"bd-65","title":"Update EXTENDING.md with UnderlyingDB() usage and best practices","description":"EXTENDING.md currently shows how to use direct sql.Open() to access the database, but doesn't mention the new UnderlyingDB() method that's the recommended way for extensions.\n\n**Update needed:**\n1. Add section showing UnderlyingDB() usage:\n ```go\n store, err := beads.NewSQLiteStorage(dbPath)\n db := store.UnderlyingDB()\n // Create extension tables using db\n ```\n\n2. Document when to use UnderlyingDB() vs direct sql.Open():\n - Use UnderlyingDB() when you want to share the storage connection\n - Use sql.Open() when you need independent connection management\n\n3. Add safety warnings (cross-reference from UnderlyingDB() docs):\n - Don't close the DB\n - Don't modify pool settings\n - Keep transactions short\n\n4. Update the VC example to show UnderlyingDB() pattern\n\n5. Explain beads.Storage.UnderlyingDB() in the API section","status":"open","priority":1,"issue_type":"task","created_at":"2025-10-22T17:07:56.820056-07:00","updated_at":"2025-10-22T17:07:56.820056-07:00","dependencies":[{"issue_id":"bd-65","depends_on_id":"bd-57","type":"discovered-from","created_at":"2025-10-22T17:07:56.822413-07:00","created_by":"daemon"}]}
|
||||||
|
{"id":"bd-66","title":"Consider adding UnderlyingConn(ctx) for safer scoped DB access","description":"Currently UnderlyingDB() returns *sql.DB which is correct for most uses, but for extension migrations/DDL, a scoped connection might be safer.\n\n**Proposal:** Add optional UnderlyingConn(ctx) (*sql.Conn, error) method that:\n- Returns a scoped connection via s.db.Conn(ctx)\n- Encourages lifetime-bounded usage\n- Reduces temptation to tune global pool settings\n- Better for one-time DDL operations like CREATE TABLE\n\n**Implementation:**\n```go\n// UnderlyingConn returns a single connection from the pool for scoped use\n// Useful for migrations and DDL. Close the connection when done.\nfunc (s *SQLiteStorage) UnderlyingConn(ctx context.Context) (*sql.Conn, error) {\n return s.db.Conn(ctx)\n}\n```\n\n**Benefits:**\n- Safer for migrations (explicit scope)\n- Complements UnderlyingDB() for different use cases\n- Low implementation cost\n\n**Trade-off:** Adds another method to maintain, but Oracle considers this balanced compromise between safety and flexibility.\n\n**Decision:** This is optional - evaluate based on VC's actual usage patterns.","status":"open","priority":3,"issue_type":"feature","created_at":"2025-10-22T17:07:56.832638-07:00","updated_at":"2025-10-22T17:07:56.832638-07:00","dependencies":[{"issue_id":"bd-66","depends_on_id":"bd-57","type":"related","created_at":"2025-10-22T17:07:56.835844-07:00","created_by":"daemon"}]}
|
||||||
{"id":"bd-7","title":"Write tests for merge functionality","description":"Unit tests: validation, merge logic, data integrity. Integration tests: end-to-end workflow, export/import. Edge case tests: chains, circular refs, epics.","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-21T23:53:44.31362-07:00","updated_at":"2025-10-22T11:56:36.523309-07:00","closed_at":"2025-10-22T01:07:04.72062-07:00"}
|
{"id":"bd-7","title":"Write tests for merge functionality","description":"Unit tests: validation, merge logic, data integrity. Integration tests: end-to-end workflow, export/import. Edge case tests: chains, circular refs, epics.","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-21T23:53:44.31362-07:00","updated_at":"2025-10-22T11:56:36.523309-07:00","closed_at":"2025-10-22T01:07:04.72062-07:00"}
|
||||||
{"id":"bd-8","title":"Improve error handling in dependency removal during remapping","description":"In updateDependencyReferences(), RemoveDependency errors are caught and ignored with continue (line 392). Comment says 'if dependency doesn't exist' but this catches ALL errors including real failures. Should check error type with errors.Is(err, ErrDependencyNotFound) and only ignore not-found errors, returning other errors properly.","status":"closed","priority":3,"issue_type":"bug","created_at":"2025-10-21T23:53:44.31362-07:00","updated_at":"2025-10-22T11:56:36.523529-07:00","closed_at":"2025-10-18T09:41:18.209717-07:00"}
|
{"id":"bd-8","title":"Improve error handling in dependency removal during remapping","description":"In updateDependencyReferences(), RemoveDependency errors are caught and ignored with continue (line 392). Comment says 'if dependency doesn't exist' but this catches ALL errors including real failures. Should check error type with errors.Is(err, ErrDependencyNotFound) and only ignore not-found errors, returning other errors properly.","status":"closed","priority":3,"issue_type":"bug","created_at":"2025-10-21T23:53:44.31362-07:00","updated_at":"2025-10-22T11:56:36.523529-07:00","closed_at":"2025-10-18T09:41:18.209717-07:00"}
|
||||||
{"id":"bd-9","title":"Test issue 2","description":"","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-21T23:53:44.31362-07:00","updated_at":"2025-10-22T11:56:36.523753-07:00","closed_at":"2025-10-21T22:06:41.257019-07:00","labels":["test-label"]}
|
{"id":"bd-9","title":"Test issue 2","description":"","status":"closed","priority":1,"issue_type":"task","created_at":"2025-10-21T23:53:44.31362-07:00","updated_at":"2025-10-22T11:56:36.523753-07:00","closed_at":"2025-10-21T22:06:41.257019-07:00","labels":["test-label"]}
|
||||||
|
|||||||
10
go.mod
10
go.mod
@@ -2,17 +2,22 @@ module github.com/steveyegge/beads
|
|||||||
|
|
||||||
go 1.24.0
|
go 1.24.0
|
||||||
|
|
||||||
|
toolchain go1.24.2
|
||||||
|
|
||||||
require (
|
require (
|
||||||
github.com/anthropics/anthropic-sdk-go v1.14.0
|
github.com/anthropics/anthropic-sdk-go v1.14.0
|
||||||
github.com/fatih/color v1.18.0
|
github.com/fatih/color v1.18.0
|
||||||
|
github.com/google/uuid v1.6.0
|
||||||
github.com/spf13/cobra v1.10.1
|
github.com/spf13/cobra v1.10.1
|
||||||
|
golang.org/x/mod v0.29.0
|
||||||
|
golang.org/x/sys v0.36.0
|
||||||
|
gopkg.in/natefinch/lumberjack.v2 v2.2.1
|
||||||
modernc.org/sqlite v1.38.2
|
modernc.org/sqlite v1.38.2
|
||||||
rsc.io/script v0.0.2
|
rsc.io/script v0.0.2
|
||||||
)
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||||
github.com/google/uuid v1.6.0 // indirect
|
|
||||||
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
||||||
github.com/mattn/go-colorable v0.1.13 // indirect
|
github.com/mattn/go-colorable v0.1.13 // indirect
|
||||||
github.com/mattn/go-isatty v0.0.20 // indirect
|
github.com/mattn/go-isatty v0.0.20 // indirect
|
||||||
@@ -24,10 +29,7 @@ require (
|
|||||||
github.com/tidwall/pretty v1.2.1 // indirect
|
github.com/tidwall/pretty v1.2.1 // indirect
|
||||||
github.com/tidwall/sjson v1.2.5 // indirect
|
github.com/tidwall/sjson v1.2.5 // indirect
|
||||||
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b // indirect
|
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b // indirect
|
||||||
golang.org/x/mod v0.29.0 // indirect
|
|
||||||
golang.org/x/sys v0.36.0 // indirect
|
|
||||||
golang.org/x/tools v0.37.0 // indirect
|
golang.org/x/tools v0.37.0 // indirect
|
||||||
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
|
|
||||||
modernc.org/libc v1.66.3 // indirect
|
modernc.org/libc v1.66.3 // indirect
|
||||||
modernc.org/mathutil v1.7.1 // indirect
|
modernc.org/mathutil v1.7.1 // indirect
|
||||||
modernc.org/memory v1.11.0 // indirect
|
modernc.org/memory v1.11.0 // indirect
|
||||||
|
|||||||
9
go.sum
9
go.sum
@@ -44,21 +44,14 @@ github.com/tidwall/sjson v1.2.5 h1:kLy8mja+1c9jlljvWTlSazM7cKDRfJuR/bOJhcY5NcY=
|
|||||||
github.com/tidwall/sjson v1.2.5/go.mod h1:Fvgq9kS/6ociJEDnK0Fk1cpYF4FIW6ZF7LAe+6jwd28=
|
github.com/tidwall/sjson v1.2.5/go.mod h1:Fvgq9kS/6ociJEDnK0Fk1cpYF4FIW6ZF7LAe+6jwd28=
|
||||||
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b h1:M2rDM6z3Fhozi9O7NWsxAkg/yqS/lQJ6PmkyIV3YP+o=
|
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b h1:M2rDM6z3Fhozi9O7NWsxAkg/yqS/lQJ6PmkyIV3YP+o=
|
||||||
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b/go.mod h1:3//PLf8L/X+8b4vuAfHzxeRUl04Adcb341+IGKfnqS8=
|
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b/go.mod h1:3//PLf8L/X+8b4vuAfHzxeRUl04Adcb341+IGKfnqS8=
|
||||||
golang.org/x/mod v0.25.0 h1:n7a+ZbQKQA/Ysbyb0/6IbB1H/X41mKgbhfv7AfG/44w=
|
|
||||||
golang.org/x/mod v0.25.0/go.mod h1:IXM97Txy2VM4PJ3gI61r1YEk/gAj6zAHN3AdZt6S9Ww=
|
|
||||||
golang.org/x/mod v0.29.0 h1:HV8lRxZC4l2cr3Zq1LvtOsi/ThTgWnUk/y64QSs8GwA=
|
golang.org/x/mod v0.29.0 h1:HV8lRxZC4l2cr3Zq1LvtOsi/ThTgWnUk/y64QSs8GwA=
|
||||||
golang.org/x/mod v0.29.0/go.mod h1:NyhrlYXJ2H4eJiRy/WDBO6HMqZQ6q9nk4JzS3NuCK+w=
|
golang.org/x/mod v0.29.0/go.mod h1:NyhrlYXJ2H4eJiRy/WDBO6HMqZQ6q9nk4JzS3NuCK+w=
|
||||||
golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw=
|
|
||||||
golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
|
|
||||||
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
|
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
|
||||||
|
golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
|
||||||
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
|
|
||||||
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
|
||||||
golang.org/x/sys v0.36.0 h1:KVRy2GtZBrk1cBYA7MKu5bEZFxQk4NIDV6RLVcC8o0k=
|
golang.org/x/sys v0.36.0 h1:KVRy2GtZBrk1cBYA7MKu5bEZFxQk4NIDV6RLVcC8o0k=
|
||||||
golang.org/x/sys v0.36.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
golang.org/x/sys v0.36.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
||||||
golang.org/x/tools v0.34.0 h1:qIpSLOxeCYGg9TrcJokLBG4KFA6d795g0xkBkiESGlo=
|
|
||||||
golang.org/x/tools v0.34.0/go.mod h1:pAP9OwEaY1CAW3HOmg3hLZC5Z0CCmzjAF2UQMSqNARg=
|
|
||||||
golang.org/x/tools v0.37.0 h1:DVSRzp7FwePZW356yEAChSdNcQo6Nsp+fex1SUW09lE=
|
golang.org/x/tools v0.37.0 h1:DVSRzp7FwePZW356yEAChSdNcQo6Nsp+fex1SUW09lE=
|
||||||
golang.org/x/tools v0.37.0/go.mod h1:MBN5QPQtLMHVdvsbtarmTNukZDdgwdwlO5qGacAzF0w=
|
golang.org/x/tools v0.37.0/go.mod h1:MBN5QPQtLMHVdvsbtarmTNukZDdgwdwlO5qGacAzF0w=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
|
|||||||
@@ -1899,3 +1899,9 @@ func (s *SQLiteStorage) Close() error {
|
|||||||
func (s *SQLiteStorage) Path() string {
|
func (s *SQLiteStorage) Path() string {
|
||||||
return s.dbPath
|
return s.dbPath
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// UnderlyingDB returns the underlying *sql.DB connection
|
||||||
|
// This allows extensions (like VC) to create their own tables in the same database
|
||||||
|
func (s *SQLiteStorage) UnderlyingDB() *sql.DB {
|
||||||
|
return s.db
|
||||||
|
}
|
||||||
|
|||||||
289
internal/storage/sqlite/underlying_db_test.go
Normal file
289
internal/storage/sqlite/underlying_db_test.go
Normal file
@@ -0,0 +1,289 @@
|
|||||||
|
package sqlite
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/steveyegge/beads/internal/types"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestUnderlyingDB_BasicAccess tests that UnderlyingDB returns a usable connection
|
||||||
|
func TestUnderlyingDB_BasicAccess(t *testing.T) {
|
||||||
|
tmpDir, err := os.MkdirTemp("", "beads-underlying-test-*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tmpDir)
|
||||||
|
|
||||||
|
dbPath := filepath.Join(tmpDir, "test.db")
|
||||||
|
store, err := New(dbPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create storage: %v", err)
|
||||||
|
}
|
||||||
|
defer store.Close()
|
||||||
|
|
||||||
|
// Get underlying DB
|
||||||
|
db := store.UnderlyingDB()
|
||||||
|
if db == nil {
|
||||||
|
t.Fatal("UnderlyingDB() returned nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify we can query it
|
||||||
|
var count int
|
||||||
|
err = db.QueryRow("SELECT COUNT(*) FROM issues").Scan(&count)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to query via UnderlyingDB: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if count != 0 {
|
||||||
|
t.Errorf("Expected 0 issues, got %d", count)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUnderlyingDB_CreateExtensionTable tests creating a VC-style extension table
|
||||||
|
func TestUnderlyingDB_CreateExtensionTable(t *testing.T) {
|
||||||
|
tmpDir, err := os.MkdirTemp("", "beads-extension-test-*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tmpDir)
|
||||||
|
|
||||||
|
dbPath := filepath.Join(tmpDir, "test.db")
|
||||||
|
store, err := New(dbPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create storage: %v", err)
|
||||||
|
}
|
||||||
|
defer store.Close()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
// Create a test issue first
|
||||||
|
issue := &types.Issue{
|
||||||
|
Title: "Test issue",
|
||||||
|
Description: "For extension testing",
|
||||||
|
Status: types.StatusOpen,
|
||||||
|
Priority: 1,
|
||||||
|
IssueType: types.TypeTask,
|
||||||
|
}
|
||||||
|
if err := store.CreateIssue(ctx, issue, "test"); err != nil {
|
||||||
|
t.Fatalf("Failed to create issue: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get underlying DB and create extension table
|
||||||
|
db := store.UnderlyingDB()
|
||||||
|
|
||||||
|
schema := `
|
||||||
|
CREATE TABLE IF NOT EXISTS vc_executions (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
issue_id TEXT NOT NULL,
|
||||||
|
status TEXT NOT NULL,
|
||||||
|
agent_id TEXT,
|
||||||
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_vc_executions_issue ON vc_executions(issue_id);
|
||||||
|
`
|
||||||
|
|
||||||
|
if _, err := db.Exec(schema); err != nil {
|
||||||
|
t.Fatalf("Failed to create extension table: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Insert a row linking to our issue
|
||||||
|
result, err := db.Exec(`
|
||||||
|
INSERT INTO vc_executions (issue_id, status, agent_id)
|
||||||
|
VALUES (?, ?, ?)
|
||||||
|
`, issue.ID, "pending", "test-agent")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to insert into extension table: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
id, _ := result.LastInsertId()
|
||||||
|
if id == 0 {
|
||||||
|
t.Error("Expected non-zero insert ID")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify FK enforcement - try to insert with invalid issue_id
|
||||||
|
_, err = db.Exec(`
|
||||||
|
INSERT INTO vc_executions (issue_id, status, agent_id)
|
||||||
|
VALUES (?, ?, ?)
|
||||||
|
`, "invalid-id", "pending", "test-agent")
|
||||||
|
if err == nil {
|
||||||
|
t.Error("Expected FK constraint violation, got nil error")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Query across layers (join)
|
||||||
|
var title string
|
||||||
|
var status string
|
||||||
|
err = db.QueryRow(`
|
||||||
|
SELECT i.title, e.status
|
||||||
|
FROM issues i
|
||||||
|
JOIN vc_executions e ON i.id = e.issue_id
|
||||||
|
WHERE i.id = ?
|
||||||
|
`, issue.ID).Scan(&title, &status)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to join across layers: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if title != issue.Title {
|
||||||
|
t.Errorf("Expected title %q, got %q", issue.Title, title)
|
||||||
|
}
|
||||||
|
if status != "pending" {
|
||||||
|
t.Errorf("Expected status 'pending', got %q", status)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUnderlyingDB_ConcurrentAccess tests concurrent access to UnderlyingDB
|
||||||
|
func TestUnderlyingDB_ConcurrentAccess(t *testing.T) {
|
||||||
|
tmpDir, err := os.MkdirTemp("", "beads-concurrent-test-*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tmpDir)
|
||||||
|
|
||||||
|
dbPath := filepath.Join(tmpDir, "test.db")
|
||||||
|
store, err := New(dbPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create storage: %v", err)
|
||||||
|
}
|
||||||
|
defer store.Close()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
db := store.UnderlyingDB()
|
||||||
|
|
||||||
|
// Create some test issues
|
||||||
|
for i := 0; i < 10; i++ {
|
||||||
|
issue := &types.Issue{
|
||||||
|
Title: "Test issue",
|
||||||
|
Status: types.StatusOpen,
|
||||||
|
Priority: 1,
|
||||||
|
IssueType: types.TypeTask,
|
||||||
|
}
|
||||||
|
if err := store.CreateIssue(ctx, issue, "test"); err != nil {
|
||||||
|
t.Fatalf("Failed to create issue: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Spawn concurrent goroutines using both storage and raw DB
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
errors := make(chan error, 50)
|
||||||
|
|
||||||
|
// 10 goroutines querying via UnderlyingDB
|
||||||
|
for i := 0; i < 10; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
var count int
|
||||||
|
if err := db.QueryRow("SELECT COUNT(*) FROM issues").Scan(&count); err != nil {
|
||||||
|
errors <- err
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
// 10 goroutines using storage methods
|
||||||
|
for i := 0; i < 10; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
if _, err := store.SearchIssues(ctx, "", types.IssueFilter{}); err != nil {
|
||||||
|
errors <- err
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Wait for completion
|
||||||
|
wg.Wait()
|
||||||
|
close(errors)
|
||||||
|
|
||||||
|
// Check for errors
|
||||||
|
for err := range errors {
|
||||||
|
t.Errorf("Concurrent access error: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUnderlyingDB_AfterClose tests behavior after storage is closed
|
||||||
|
func TestUnderlyingDB_AfterClose(t *testing.T) {
|
||||||
|
tmpDir, err := os.MkdirTemp("", "beads-close-test-*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tmpDir)
|
||||||
|
|
||||||
|
dbPath := filepath.Join(tmpDir, "test.db")
|
||||||
|
store, err := New(dbPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create storage: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get DB reference before closing
|
||||||
|
db := store.UnderlyingDB()
|
||||||
|
|
||||||
|
// Close storage
|
||||||
|
if err := store.Close(); err != nil {
|
||||||
|
t.Fatalf("Failed to close storage: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try to use DB - should fail
|
||||||
|
var count int
|
||||||
|
err = db.QueryRow("SELECT COUNT(*) FROM issues").Scan(&count)
|
||||||
|
if err == nil {
|
||||||
|
t.Error("Expected error after close, got nil")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUnderlyingDB_LongTxDoesNotDeadlock tests that long read tx doesn't block writes forever
|
||||||
|
func TestUnderlyingDB_LongTxDoesNotDeadlock(t *testing.T) {
|
||||||
|
tmpDir, err := os.MkdirTemp("", "beads-tx-test-*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tmpDir)
|
||||||
|
|
||||||
|
dbPath := filepath.Join(tmpDir, "test.db")
|
||||||
|
store, err := New(dbPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create storage: %v", err)
|
||||||
|
}
|
||||||
|
defer store.Close()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
db := store.UnderlyingDB()
|
||||||
|
|
||||||
|
// Start a long-running read transaction
|
||||||
|
tx, err := db.BeginTx(ctx, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to begin tx: %v", err)
|
||||||
|
}
|
||||||
|
defer tx.Rollback()
|
||||||
|
|
||||||
|
// Query in the transaction
|
||||||
|
var count int
|
||||||
|
if err := tx.QueryRow("SELECT COUNT(*) FROM issues").Scan(&count); err != nil {
|
||||||
|
t.Fatalf("Failed to query in tx: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try to create an issue via storage (should not deadlock due to WAL + busy_timeout)
|
||||||
|
done := make(chan error, 1)
|
||||||
|
go func() {
|
||||||
|
issue := &types.Issue{
|
||||||
|
Title: "Test during long tx",
|
||||||
|
Status: types.StatusOpen,
|
||||||
|
Priority: 1,
|
||||||
|
IssueType: types.TypeTask,
|
||||||
|
}
|
||||||
|
done <- store.CreateIssue(ctx, issue, "test")
|
||||||
|
}()
|
||||||
|
|
||||||
|
// Wait with timeout
|
||||||
|
select {
|
||||||
|
case err := <-done:
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("CreateIssue failed during long tx: %v", err)
|
||||||
|
}
|
||||||
|
case <-time.After(5 * time.Second):
|
||||||
|
t.Error("CreateIssue deadlocked or timed out")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -3,6 +3,7 @@ package storage
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
|
||||||
"github.com/steveyegge/beads/internal/types"
|
"github.com/steveyegge/beads/internal/types"
|
||||||
)
|
)
|
||||||
@@ -72,6 +73,12 @@ type Storage interface {
|
|||||||
|
|
||||||
// Database path (for daemon validation)
|
// Database path (for daemon validation)
|
||||||
Path() string
|
Path() string
|
||||||
|
|
||||||
|
// UnderlyingDB returns the underlying *sql.DB connection
|
||||||
|
// This is provided for extensions (like VC) that need to create their own tables
|
||||||
|
// in the same database. Extensions should use foreign keys to reference core tables.
|
||||||
|
// WARNING: Direct database access bypasses the storage layer. Use with caution.
|
||||||
|
UnderlyingDB() *sql.DB
|
||||||
}
|
}
|
||||||
|
|
||||||
// Config holds database configuration
|
// Config holds database configuration
|
||||||
|
|||||||
Reference in New Issue
Block a user