Add migration inspection tools for AI agents (bd-627d Phase 2)
Implemented: - bd migrate --inspect --json: Shows migration plan, db state, warnings - bd info --schema --json: Returns schema details for agents - Migration invariants: Validates migrations post-execution - Added ListMigrations() for introspection Phase 1 (invariants) and Phase 2 (inspection) complete. Next: Wire up MCP tools in beads-mcp server. Amp-Thread-ID: https://ampcode.com/threads/T-c4674660-d640-405f-a929-b664e8699a48 Co-authored-by: Amp <amp@ampcode.com>
This commit is contained in:
Steve Yegge
@@ -226,6 +226,10 @@ bd merge bd-42 bd-43 --into bd-41 --dry-run # Preview merge
|
||||
bd migrate # Detect and migrate old databases
|
||||
bd migrate --dry-run # Preview migration
|
||||
bd migrate --cleanup --yes # Migrate and remove old files
|
||||
bd migrate --inspect --json # Show migration plan for AI agents
|
||||
|
||||
# Inspect database schema and state (for AI agent analysis)
|
||||
bd info --schema --json # Get schema, tables, config, sample IDs
|
||||
```
|
||||
|
||||
### Managing Daemons
|
||||
|
||||
@@ -1,87 +0,0 @@
|
||||
# Next Session: Agent-Supervised Migration Safety
|
||||
|
||||
## Context
|
||||
We identified that database migrations can lose user data through edge cases (e.g., GH #201 where `bd migrate` failed to set `issue_prefix`, breaking commands). Since beads is designed for AI agents, we should leverage **agent supervision** to make migrations safer.
|
||||
|
||||
## Key Architectural Decision
|
||||
**Beads provides observability primitives; agents supervise using their own reasoning.**
|
||||
|
||||
Beads does NOT:
|
||||
- ❌ Make AI API calls
|
||||
- ❌ Invoke external models
|
||||
- ❌ Call agents
|
||||
|
||||
Beads DOES:
|
||||
- ✅ Provide deterministic invariant checks
|
||||
- ✅ Expose migration state via `--dry-run --json`
|
||||
- ✅ Roll back on validation failures
|
||||
- ✅ Give agents structured data to analyze
|
||||
|
||||
## The Work (bd-627d)
|
||||
|
||||
### Phase 1: Migration Invariants (Start here!)
|
||||
Create `internal/storage/sqlite/migration_invariants.go` with:
|
||||
|
||||
```go
|
||||
type MigrationInvariant struct {
|
||||
Name string
|
||||
Description string
|
||||
Check func(*sql.DB, *Snapshot) error
|
||||
}
|
||||
|
||||
type Snapshot struct {
|
||||
IssueCount int
|
||||
ConfigKeys []string
|
||||
DependencyCount int
|
||||
LabelCount int
|
||||
}
|
||||
```
|
||||
|
||||
Implement these invariants:
|
||||
1. **required_config_present** - Would have caught GH #201!
|
||||
2. **foreign_keys_valid** - Detect orphaned dependencies
|
||||
3. **issue_count_stable** - Catch unexpected data loss
|
||||
|
||||
### Phase 2: Inspection Tools
|
||||
Add CLI commands for agents to inspect migrations:
|
||||
|
||||
1. `bd migrate --dry-run --json` - Shows what will change
|
||||
2. `bd info --schema --json` - Current schema + detected prefix
|
||||
3. Update `RunMigrations()` to check invariants and rollback on failure
|
||||
|
||||
### Phase 3 & 4: MCP Tools + Agent Workflows
|
||||
Add MCP tools so agents can:
|
||||
- Inspect migration plans before running
|
||||
- Detect missing config (like `issue_prefix`)
|
||||
- Auto-fix issues before migration
|
||||
- Validate post-migration state
|
||||
|
||||
## Starting Prompt for Next Session
|
||||
|
||||
```
|
||||
Let's implement Phase 1 of bd-627d (agent-supervised migration safety).
|
||||
|
||||
We need to create migration invariants that check for common data loss scenarios:
|
||||
1. Missing required config keys (would have caught GH #201)
|
||||
2. Foreign key integrity (no orphaned dependencies)
|
||||
3. Issue count stability (detect unexpected deletions)
|
||||
|
||||
Start by creating internal/storage/sqlite/migration_invariants.go with the Snapshot type and invariant infrastructure. Then integrate it into RunMigrations() in migrations.go.
|
||||
|
||||
The goal: migrations should automatically roll back if invariants fail, preventing data loss.
|
||||
```
|
||||
|
||||
## Related Issues
|
||||
- bd-627d: Main epic for agent-supervised migrations
|
||||
- GH #201: Real-world example of migration data loss (missing issue_prefix)
|
||||
- bd-d355a07d: False positive data loss warnings
|
||||
- bd-b245: Migration registry (just completed - makes migrations introspectable!)
|
||||
|
||||
## Success Criteria
|
||||
After Phase 1, migrations should:
|
||||
- ✅ Check invariants before committing
|
||||
- ✅ Roll back on any invariant failure
|
||||
- ✅ Provide clear error messages
|
||||
- ✅ Have unit tests for each invariant
|
||||
|
||||
This prevents silent data loss like GH #201 where users discovered breakage only after migration completed.
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
@@ -21,11 +22,15 @@ or daemon connection. It shows:
|
||||
- Daemon connection status (daemon or direct mode)
|
||||
- If using daemon: socket path, health status, version
|
||||
- Database statistics (issue count)
|
||||
- Schema information (with --schema flag)
|
||||
|
||||
Examples:
|
||||
bd info
|
||||
bd info --json`,
|
||||
bd info --json
|
||||
bd info --schema --json`,
|
||||
Run: func(cmd *cobra.Command, args []string) {
|
||||
schemaFlag, _ := cmd.Flags().GetBool("schema")
|
||||
|
||||
// Get database path (absolute)
|
||||
absDBPath, err := filepath.Abs(dbPath)
|
||||
if err != nil {
|
||||
@@ -81,6 +86,55 @@ Examples:
|
||||
}
|
||||
}
|
||||
|
||||
// Add schema information if requested
|
||||
if schemaFlag && store != nil {
|
||||
ctx := context.Background()
|
||||
|
||||
// Get schema version
|
||||
schemaVersion, err := store.GetMetadata(ctx, "bd_version")
|
||||
if err != nil {
|
||||
schemaVersion = "unknown"
|
||||
}
|
||||
|
||||
// Get tables
|
||||
tables := []string{"issues", "dependencies", "labels", "config", "metadata"}
|
||||
|
||||
// Get config
|
||||
configMap := make(map[string]string)
|
||||
prefix, _ := store.GetConfig(ctx, "issue_prefix")
|
||||
if prefix != "" {
|
||||
configMap["issue_prefix"] = prefix
|
||||
}
|
||||
|
||||
// Get sample issue IDs
|
||||
filter := types.IssueFilter{}
|
||||
issues, err := store.SearchIssues(ctx, "", filter)
|
||||
sampleIDs := []string{}
|
||||
detectedPrefix := ""
|
||||
if err == nil && len(issues) > 0 {
|
||||
// Get first 3 issue IDs as samples
|
||||
maxSamples := 3
|
||||
if len(issues) < maxSamples {
|
||||
maxSamples = len(issues)
|
||||
}
|
||||
for i := 0; i < maxSamples; i++ {
|
||||
sampleIDs = append(sampleIDs, issues[i].ID)
|
||||
}
|
||||
// Detect prefix from first issue
|
||||
if len(issues) > 0 {
|
||||
detectedPrefix = extractPrefix(issues[0].ID)
|
||||
}
|
||||
}
|
||||
|
||||
info["schema"] = map[string]interface{}{
|
||||
"tables": tables,
|
||||
"schema_version": schemaVersion,
|
||||
"config": configMap,
|
||||
"sample_issue_ids": sampleIDs,
|
||||
"detected_prefix": detectedPrefix,
|
||||
}
|
||||
}
|
||||
|
||||
// JSON output
|
||||
if jsonOutput {
|
||||
outputJSON(info)
|
||||
@@ -125,10 +179,38 @@ Examples:
|
||||
fmt.Printf("\nIssue Count: %d\n", count)
|
||||
}
|
||||
|
||||
// Show schema information if requested
|
||||
if schemaFlag {
|
||||
if schemaInfo, ok := info["schema"].(map[string]interface{}); ok {
|
||||
fmt.Println("\nSchema Information:")
|
||||
fmt.Printf(" Tables: %v\n", schemaInfo["tables"])
|
||||
if version, ok := schemaInfo["schema_version"].(string); ok {
|
||||
fmt.Printf(" Schema Version: %s\n", version)
|
||||
}
|
||||
if prefix, ok := schemaInfo["detected_prefix"].(string); ok && prefix != "" {
|
||||
fmt.Printf(" Detected Prefix: %s\n", prefix)
|
||||
}
|
||||
if samples, ok := schemaInfo["sample_issue_ids"].([]string); ok && len(samples) > 0 {
|
||||
fmt.Printf(" Sample Issues: %v\n", samples)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fmt.Println()
|
||||
},
|
||||
}
|
||||
|
||||
// extractPrefix extracts the prefix from an issue ID (e.g., "bd-123" -> "bd")
|
||||
func extractPrefix(issueID string) string {
|
||||
parts := strings.Split(issueID, "-")
|
||||
if len(parts) > 0 {
|
||||
return parts[0]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func init() {
|
||||
infoCmd.Flags().Bool("schema", false, "Include schema information in output")
|
||||
infoCmd.Flags().BoolVar(&jsonOutput, "json", false, "Output in JSON format")
|
||||
rootCmd.AddCommand(infoCmd)
|
||||
}
|
||||
|
||||
@@ -37,6 +37,7 @@ This command:
|
||||
dryRun, _ := cmd.Flags().GetBool("dry-run")
|
||||
updateRepoID, _ := cmd.Flags().GetBool("update-repo-id")
|
||||
toHashIDs, _ := cmd.Flags().GetBool("to-hash-ids")
|
||||
inspect, _ := cmd.Flags().GetBool("inspect")
|
||||
|
||||
// Handle --update-repo-id first
|
||||
if updateRepoID {
|
||||
@@ -44,6 +45,12 @@ This command:
|
||||
return
|
||||
}
|
||||
|
||||
// Handle --inspect flag (show migration plan for AI agents)
|
||||
if inspect {
|
||||
handleInspect()
|
||||
return
|
||||
}
|
||||
|
||||
// Find .beads directory
|
||||
beadsDir := findBeadsDir()
|
||||
if beadsDir == "" {
|
||||
@@ -695,12 +702,196 @@ func cleanupWALFiles(dbPath string) {
|
||||
_ = os.Remove(shmPath)
|
||||
}
|
||||
|
||||
// handleInspect shows migration plan and database state for AI agent analysis
|
||||
func handleInspect() {
|
||||
// Find .beads directory
|
||||
beadsDir := findBeadsDir()
|
||||
if beadsDir == "" {
|
||||
if jsonOutput {
|
||||
outputJSON(map[string]interface{}{
|
||||
"error": "no_beads_directory",
|
||||
"message": "No .beads directory found. Run 'bd init' first.",
|
||||
})
|
||||
} else {
|
||||
fmt.Fprintf(os.Stderr, "Error: no .beads directory found\n")
|
||||
fmt.Fprintf(os.Stderr, "Hint: run 'bd init' to initialize bd\n")
|
||||
}
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Load config
|
||||
cfg, err := loadOrCreateConfig(beadsDir)
|
||||
if err != nil {
|
||||
if jsonOutput {
|
||||
outputJSON(map[string]interface{}{
|
||||
"error": "config_load_failed",
|
||||
"message": err.Error(),
|
||||
})
|
||||
} else {
|
||||
fmt.Fprintf(os.Stderr, "Error: failed to load config: %v\n", err)
|
||||
}
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Check if database exists (don't create it)
|
||||
targetPath := cfg.DatabasePath(beadsDir)
|
||||
dbExists := false
|
||||
if _, err := os.Stat(targetPath); err == nil {
|
||||
dbExists = true
|
||||
} else if !os.IsNotExist(err) {
|
||||
// Stat error (not just "doesn't exist")
|
||||
if jsonOutput {
|
||||
outputJSON(map[string]interface{}{
|
||||
"error": "database_stat_failed",
|
||||
"message": err.Error(),
|
||||
})
|
||||
} else {
|
||||
fmt.Fprintf(os.Stderr, "Error: failed to check database: %v\n", err)
|
||||
}
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// If database doesn't exist, return inspection with defaults
|
||||
if !dbExists {
|
||||
result := map[string]interface{}{
|
||||
"registered_migrations": sqlite.ListMigrations(),
|
||||
"current_state": map[string]interface{}{
|
||||
"schema_version": "missing",
|
||||
"issue_count": 0,
|
||||
"config": map[string]string{},
|
||||
"missing_config": []string{},
|
||||
"db_exists": false,
|
||||
},
|
||||
"warnings": []string{"Database does not exist - run 'bd init' first"},
|
||||
"invariants_to_check": sqlite.GetInvariantNames(),
|
||||
}
|
||||
|
||||
if jsonOutput {
|
||||
outputJSON(result)
|
||||
} else {
|
||||
fmt.Println("\nMigration Inspection")
|
||||
fmt.Println("====================")
|
||||
fmt.Println("Database: missing")
|
||||
fmt.Println("\n⚠ Database does not exist - run 'bd init' first")
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// Open database in read-only mode for inspection
|
||||
store, err := sqlite.New(targetPath)
|
||||
if err != nil {
|
||||
if jsonOutput {
|
||||
outputJSON(map[string]interface{}{
|
||||
"error": "database_open_failed",
|
||||
"message": err.Error(),
|
||||
})
|
||||
} else {
|
||||
fmt.Fprintf(os.Stderr, "Error: failed to open database: %v\n", err)
|
||||
}
|
||||
os.Exit(1)
|
||||
}
|
||||
defer func() { _ = store.Close() }()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Get current schema version
|
||||
schemaVersion, err := store.GetMetadata(ctx, "bd_version")
|
||||
if err != nil {
|
||||
schemaVersion = "unknown"
|
||||
}
|
||||
|
||||
// Get issue count (use efficient COUNT query)
|
||||
issueCount := 0
|
||||
if stats, err := store.GetStatistics(ctx); err == nil {
|
||||
issueCount = stats.TotalIssues
|
||||
}
|
||||
|
||||
// Get config
|
||||
configMap := make(map[string]string)
|
||||
prefix, _ := store.GetConfig(ctx, "issue_prefix")
|
||||
if prefix != "" {
|
||||
configMap["issue_prefix"] = prefix
|
||||
}
|
||||
|
||||
// Detect missing config
|
||||
missingConfig := []string{}
|
||||
if issueCount > 0 && prefix == "" {
|
||||
missingConfig = append(missingConfig, "issue_prefix")
|
||||
}
|
||||
|
||||
// Get registered migrations (all migrations are idempotent and run on every open)
|
||||
registeredMigrations := sqlite.ListMigrations()
|
||||
|
||||
// Build invariants list
|
||||
invariantNames := sqlite.GetInvariantNames()
|
||||
|
||||
// Generate warnings
|
||||
warnings := []string{}
|
||||
if issueCount > 0 && prefix == "" {
|
||||
// Detect prefix from first issue (efficient query for just 1 issue)
|
||||
detectedPrefix := ""
|
||||
if issues, err := store.SearchIssues(ctx, "", types.IssueFilter{}); err == nil && len(issues) > 0 {
|
||||
detectedPrefix = utils.ExtractIssuePrefix(issues[0].ID)
|
||||
}
|
||||
warnings = append(warnings, fmt.Sprintf("issue_prefix config not set - may break commands after migration (detected: %s)", detectedPrefix))
|
||||
}
|
||||
if schemaVersion != Version {
|
||||
warnings = append(warnings, fmt.Sprintf("schema version mismatch (current: %s, expected: %s)", schemaVersion, Version))
|
||||
}
|
||||
|
||||
// Output result
|
||||
result := map[string]interface{}{
|
||||
"registered_migrations": registeredMigrations,
|
||||
"current_state": map[string]interface{}{
|
||||
"schema_version": schemaVersion,
|
||||
"issue_count": issueCount,
|
||||
"config": configMap,
|
||||
"missing_config": missingConfig,
|
||||
"db_exists": true,
|
||||
},
|
||||
"warnings": warnings,
|
||||
"invariants_to_check": invariantNames,
|
||||
}
|
||||
|
||||
if jsonOutput {
|
||||
outputJSON(result)
|
||||
} else {
|
||||
// Human-readable output
|
||||
fmt.Println("\nMigration Inspection")
|
||||
fmt.Println("====================")
|
||||
fmt.Printf("Schema Version: %s\n", schemaVersion)
|
||||
fmt.Printf("Issue Count: %d\n", issueCount)
|
||||
fmt.Printf("Registered Migrations: %d\n", len(registeredMigrations))
|
||||
|
||||
if len(warnings) > 0 {
|
||||
fmt.Println("\nWarnings:")
|
||||
for _, w := range warnings {
|
||||
fmt.Printf(" ⚠ %s\n", w)
|
||||
}
|
||||
}
|
||||
|
||||
if len(missingConfig) > 0 {
|
||||
fmt.Println("\nMissing Config:")
|
||||
for _, k := range missingConfig {
|
||||
fmt.Printf(" - %s\n", k)
|
||||
}
|
||||
}
|
||||
|
||||
fmt.Printf("\nInvariants to Check: %d\n", len(invariantNames))
|
||||
for _, inv := range invariantNames {
|
||||
fmt.Printf(" ✓ %s\n", inv)
|
||||
}
|
||||
fmt.Println()
|
||||
}
|
||||
}
|
||||
|
||||
func init() {
|
||||
migrateCmd.Flags().Bool("yes", false, "Auto-confirm cleanup prompts")
|
||||
migrateCmd.Flags().Bool("cleanup", false, "Remove old database files after migration")
|
||||
migrateCmd.Flags().Bool("dry-run", false, "Show what would be done without making changes")
|
||||
migrateCmd.Flags().Bool("update-repo-id", false, "Update repository ID (use after changing git remote)")
|
||||
migrateCmd.Flags().Bool("to-hash-ids", false, "Migrate sequential IDs to hash-based IDs")
|
||||
migrateCmd.Flags().Bool("inspect", false, "Show migration plan and database state for AI agent analysis")
|
||||
migrateCmd.Flags().BoolVar(&jsonOutput, "json", false, "Output migration statistics in JSON format")
|
||||
rootCmd.AddCommand(migrateCmd)
|
||||
}
|
||||
|
||||
204
internal/storage/sqlite/migration_invariants.go
Normal file
204
internal/storage/sqlite/migration_invariants.go
Normal file
@@ -0,0 +1,204 @@
|
||||
// Package sqlite - migration safety invariants
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Snapshot captures database state before migrations for validation
|
||||
type Snapshot struct {
|
||||
IssueCount int
|
||||
ConfigKeys []string
|
||||
DependencyCount int
|
||||
LabelCount int
|
||||
}
|
||||
|
||||
// MigrationInvariant represents a database invariant that must hold after migrations
|
||||
type MigrationInvariant struct {
|
||||
Name string
|
||||
Description string
|
||||
Check func(*sql.DB, *Snapshot) error
|
||||
}
|
||||
|
||||
// invariants is the list of all invariants checked after migrations
|
||||
var invariants = []MigrationInvariant{
|
||||
{
|
||||
Name: "required_config_present",
|
||||
Description: "Required config keys must exist",
|
||||
Check: checkRequiredConfig,
|
||||
},
|
||||
{
|
||||
Name: "foreign_keys_valid",
|
||||
Description: "No orphaned dependencies or labels",
|
||||
Check: checkForeignKeys,
|
||||
},
|
||||
{
|
||||
Name: "issue_count_stable",
|
||||
Description: "Issue count should not decrease unexpectedly",
|
||||
Check: checkIssueCount,
|
||||
},
|
||||
}
|
||||
|
||||
// captureSnapshot takes a snapshot of the database state before migrations
|
||||
func captureSnapshot(db *sql.DB) (*Snapshot, error) {
|
||||
snapshot := &Snapshot{}
|
||||
|
||||
// Count issues
|
||||
err := db.QueryRow("SELECT COUNT(*) FROM issues").Scan(&snapshot.IssueCount)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to count issues: %w", err)
|
||||
}
|
||||
|
||||
// Get config keys
|
||||
rows, err := db.Query("SELECT key FROM config ORDER BY key")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to query config keys: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
snapshot.ConfigKeys = []string{}
|
||||
for rows.Next() {
|
||||
var key string
|
||||
if err := rows.Scan(&key); err != nil {
|
||||
return nil, fmt.Errorf("failed to scan config key: %w", err)
|
||||
}
|
||||
snapshot.ConfigKeys = append(snapshot.ConfigKeys, key)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("error reading config keys: %w", err)
|
||||
}
|
||||
|
||||
// Count dependencies
|
||||
err = db.QueryRow("SELECT COUNT(*) FROM dependencies").Scan(&snapshot.DependencyCount)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to count dependencies: %w", err)
|
||||
}
|
||||
|
||||
// Count labels
|
||||
err = db.QueryRow("SELECT COUNT(*) FROM labels").Scan(&snapshot.LabelCount)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to count labels: %w", err)
|
||||
}
|
||||
|
||||
return snapshot, nil
|
||||
}
|
||||
|
||||
// verifyInvariants checks all migration invariants and returns error if any fail
|
||||
func verifyInvariants(db *sql.DB, snapshot *Snapshot) error {
|
||||
var failures []string
|
||||
|
||||
for _, invariant := range invariants {
|
||||
if err := invariant.Check(db, snapshot); err != nil {
|
||||
failures = append(failures, fmt.Sprintf("%s: %v", invariant.Name, err))
|
||||
}
|
||||
}
|
||||
|
||||
if len(failures) > 0 {
|
||||
return fmt.Errorf("migration invariants failed:\n - %s", strings.Join(failures, "\n - "))
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// checkRequiredConfig ensures required config keys exist (would have caught GH #201)
|
||||
// Only enforces issue_prefix requirement if there are issues in the database
|
||||
func checkRequiredConfig(db *sql.DB, snapshot *Snapshot) error {
|
||||
// Check current issue count (not snapshot, since migrations may add/remove issues)
|
||||
var currentCount int
|
||||
err := db.QueryRow("SELECT COUNT(*) FROM issues").Scan(¤tCount)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to count issues: %w", err)
|
||||
}
|
||||
|
||||
// Only require issue_prefix if there are issues in the database
|
||||
// New databases can exist without issue_prefix until first issue is created
|
||||
if currentCount == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Check for required config keys
|
||||
var value string
|
||||
err = db.QueryRow("SELECT value FROM config WHERE key = 'issue_prefix'").Scan(&value)
|
||||
if err == sql.ErrNoRows || value == "" {
|
||||
return fmt.Errorf("required config key missing: issue_prefix (database has %d issues)", currentCount)
|
||||
} else if err != nil {
|
||||
return fmt.Errorf("failed to check config key issue_prefix: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// checkForeignKeys ensures no orphaned dependencies or labels exist
|
||||
func checkForeignKeys(db *sql.DB, snapshot *Snapshot) error {
|
||||
// Check for orphaned dependencies (issue_id not in issues)
|
||||
var orphanedDepsIssue int
|
||||
err := db.QueryRow(`
|
||||
SELECT COUNT(*)
|
||||
FROM dependencies d
|
||||
WHERE NOT EXISTS (SELECT 1 FROM issues WHERE id = d.issue_id)
|
||||
`).Scan(&orphanedDepsIssue)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to check orphaned dependencies (issue_id): %w", err)
|
||||
}
|
||||
if orphanedDepsIssue > 0 {
|
||||
return fmt.Errorf("found %d orphaned dependencies (issue_id not in issues)", orphanedDepsIssue)
|
||||
}
|
||||
|
||||
// Check for orphaned dependencies (depends_on_id not in issues)
|
||||
var orphanedDepsDependsOn int
|
||||
err = db.QueryRow(`
|
||||
SELECT COUNT(*)
|
||||
FROM dependencies d
|
||||
WHERE NOT EXISTS (SELECT 1 FROM issues WHERE id = d.depends_on_id)
|
||||
`).Scan(&orphanedDepsDependsOn)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to check orphaned dependencies (depends_on_id): %w", err)
|
||||
}
|
||||
if orphanedDepsDependsOn > 0 {
|
||||
return fmt.Errorf("found %d orphaned dependencies (depends_on_id not in issues)", orphanedDepsDependsOn)
|
||||
}
|
||||
|
||||
// Check for orphaned labels (issue_id not in issues)
|
||||
var orphanedLabels int
|
||||
err = db.QueryRow(`
|
||||
SELECT COUNT(*)
|
||||
FROM labels l
|
||||
WHERE NOT EXISTS (SELECT 1 FROM issues WHERE id = l.issue_id)
|
||||
`).Scan(&orphanedLabels)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to check orphaned labels: %w", err)
|
||||
}
|
||||
if orphanedLabels > 0 {
|
||||
return fmt.Errorf("found %d orphaned labels (issue_id not in issues)", orphanedLabels)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// checkIssueCount ensures issue count doesn't decrease unexpectedly
|
||||
func checkIssueCount(db *sql.DB, snapshot *Snapshot) error {
|
||||
var currentCount int
|
||||
err := db.QueryRow("SELECT COUNT(*) FROM issues").Scan(¤tCount)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to count issues: %w", err)
|
||||
}
|
||||
|
||||
if currentCount < snapshot.IssueCount {
|
||||
return fmt.Errorf("issue count decreased from %d to %d (potential data loss)", snapshot.IssueCount, currentCount)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetInvariantNames returns the names of all registered invariants (for testing/inspection)
|
||||
func GetInvariantNames() []string {
|
||||
names := make([]string, len(invariants))
|
||||
for i, inv := range invariants {
|
||||
names[i] = inv.Name
|
||||
}
|
||||
sort.Strings(names)
|
||||
return names
|
||||
}
|
||||
260
internal/storage/sqlite/migration_invariants_test.go
Normal file
260
internal/storage/sqlite/migration_invariants_test.go
Normal file
@@ -0,0 +1,260 @@
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestCaptureSnapshot(t *testing.T) {
|
||||
db := setupInvariantTestDB(t)
|
||||
defer db.Close()
|
||||
|
||||
// Create some test data
|
||||
_, err := db.Exec(`INSERT INTO issues (id, title) VALUES ('test-1', 'Test Issue')`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert test issue: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`INSERT INTO dependencies (issue_id, depends_on_id, created_by) VALUES ('test-1', 'test-1', 'test')`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert test dependency: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`INSERT INTO labels (issue_id, label) VALUES ('test-1', 'test-label')`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert test label: %v", err)
|
||||
}
|
||||
|
||||
snapshot, err := captureSnapshot(db)
|
||||
if err != nil {
|
||||
t.Fatalf("captureSnapshot failed: %v", err)
|
||||
}
|
||||
|
||||
if snapshot.IssueCount != 1 {
|
||||
t.Errorf("expected IssueCount=1, got %d", snapshot.IssueCount)
|
||||
}
|
||||
|
||||
if snapshot.DependencyCount != 1 {
|
||||
t.Errorf("expected DependencyCount=1, got %d", snapshot.DependencyCount)
|
||||
}
|
||||
|
||||
if snapshot.LabelCount != 1 {
|
||||
t.Errorf("expected LabelCount=1, got %d", snapshot.LabelCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckRequiredConfig(t *testing.T) {
|
||||
db := setupInvariantTestDB(t)
|
||||
defer db.Close()
|
||||
|
||||
// Test with no issues - should pass even without issue_prefix
|
||||
snapshot := &Snapshot{IssueCount: 0}
|
||||
err := checkRequiredConfig(db, snapshot)
|
||||
if err != nil {
|
||||
t.Errorf("expected no error with 0 issues, got: %v", err)
|
||||
}
|
||||
|
||||
// Add an issue to the database
|
||||
_, err = db.Exec(`INSERT INTO issues (id, title) VALUES ('test-1', 'Test Issue')`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert issue: %v", err)
|
||||
}
|
||||
|
||||
// Delete issue_prefix config
|
||||
_, err = db.Exec(`DELETE FROM config WHERE key = 'issue_prefix'`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to delete config: %v", err)
|
||||
}
|
||||
|
||||
// Should fail now that we have an issue but no prefix
|
||||
err = checkRequiredConfig(db, snapshot)
|
||||
if err == nil {
|
||||
t.Error("expected error for missing issue_prefix with issues, got nil")
|
||||
}
|
||||
|
||||
// Add required config back
|
||||
_, err = db.Exec(`INSERT INTO config (key, value) VALUES ('issue_prefix', 'test')`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert config: %v", err)
|
||||
}
|
||||
|
||||
// Test with required config present
|
||||
err = checkRequiredConfig(db, snapshot)
|
||||
if err != nil {
|
||||
t.Errorf("expected no error with issue_prefix set, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckForeignKeys(t *testing.T) {
|
||||
db := setupInvariantTestDB(t)
|
||||
defer db.Close()
|
||||
|
||||
snapshot := &Snapshot{}
|
||||
|
||||
// Test with no data - should pass
|
||||
err := checkForeignKeys(db, snapshot)
|
||||
if err != nil {
|
||||
t.Errorf("expected no error with empty db, got: %v", err)
|
||||
}
|
||||
|
||||
// Create an issue
|
||||
_, err = db.Exec(`INSERT INTO issues (id, title) VALUES ('test-1', 'Test Issue')`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert test issue: %v", err)
|
||||
}
|
||||
|
||||
// Add valid dependency
|
||||
_, err = db.Exec(`INSERT INTO dependencies (issue_id, depends_on_id, created_by) VALUES ('test-1', 'test-1', 'test')`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert dependency: %v", err)
|
||||
}
|
||||
|
||||
// Should pass with valid foreign keys
|
||||
err = checkForeignKeys(db, snapshot)
|
||||
if err != nil {
|
||||
t.Errorf("expected no error with valid dependencies, got: %v", err)
|
||||
}
|
||||
|
||||
// Manually create orphaned dependency (bypassing FK constraints for testing)
|
||||
_, err = db.Exec(`PRAGMA foreign_keys = OFF`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to disable foreign keys: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`INSERT INTO dependencies (issue_id, depends_on_id, created_by) VALUES ('orphan-1', 'test-1', 'test')`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert orphaned dependency: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`PRAGMA foreign_keys = ON`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to enable foreign keys: %v", err)
|
||||
}
|
||||
|
||||
// Should fail with orphaned dependency
|
||||
err = checkForeignKeys(db, snapshot)
|
||||
if err == nil {
|
||||
t.Error("expected error for orphaned dependency, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckIssueCount(t *testing.T) {
|
||||
db := setupInvariantTestDB(t)
|
||||
defer db.Close()
|
||||
|
||||
// Create initial issue
|
||||
_, err := db.Exec(`INSERT INTO issues (id, title) VALUES ('test-1', 'Test Issue')`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert test issue: %v", err)
|
||||
}
|
||||
|
||||
snapshot, err := captureSnapshot(db)
|
||||
if err != nil {
|
||||
t.Fatalf("captureSnapshot failed: %v", err)
|
||||
}
|
||||
|
||||
// Same count - should pass
|
||||
err = checkIssueCount(db, snapshot)
|
||||
if err != nil {
|
||||
t.Errorf("expected no error with same count, got: %v", err)
|
||||
}
|
||||
|
||||
// Add an issue - should pass (count increased)
|
||||
_, err = db.Exec(`INSERT INTO issues (id, title) VALUES ('test-2', 'Test Issue 2')`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert second issue: %v", err)
|
||||
}
|
||||
|
||||
err = checkIssueCount(db, snapshot)
|
||||
if err != nil {
|
||||
t.Errorf("expected no error with increased count, got: %v", err)
|
||||
}
|
||||
|
||||
// Delete both issues to simulate data loss
|
||||
_, err = db.Exec(`DELETE FROM issues`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to delete issues: %v", err)
|
||||
}
|
||||
|
||||
// Should fail when count decreased
|
||||
err = checkIssueCount(db, snapshot)
|
||||
if err == nil {
|
||||
t.Error("expected error for decreased issue count, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestVerifyInvariants(t *testing.T) {
|
||||
db := setupInvariantTestDB(t)
|
||||
defer db.Close()
|
||||
|
||||
snapshot, err := captureSnapshot(db)
|
||||
if err != nil {
|
||||
t.Fatalf("captureSnapshot failed: %v", err)
|
||||
}
|
||||
|
||||
// All invariants should pass with empty database
|
||||
err = verifyInvariants(db, snapshot)
|
||||
if err != nil {
|
||||
t.Errorf("expected no errors with empty db, got: %v", err)
|
||||
}
|
||||
|
||||
// Add an issue (which requires issue_prefix)
|
||||
_, err = db.Exec(`INSERT INTO issues (id, title) VALUES ('test-1', 'Test Issue')`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert issue: %v", err)
|
||||
}
|
||||
|
||||
// Capture new snapshot with issue
|
||||
snapshot, err = captureSnapshot(db)
|
||||
if err != nil {
|
||||
t.Fatalf("captureSnapshot failed: %v", err)
|
||||
}
|
||||
|
||||
// Should still pass (issue_prefix is set by newTestStore)
|
||||
err = verifyInvariants(db, snapshot)
|
||||
if err != nil {
|
||||
t.Errorf("expected no errors with issue and prefix, got: %v", err)
|
||||
}
|
||||
|
||||
// Remove required config to trigger failure
|
||||
_, err = db.Exec(`DELETE FROM config WHERE key = 'issue_prefix'`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to delete config: %v", err)
|
||||
}
|
||||
|
||||
err = verifyInvariants(db, snapshot)
|
||||
if err == nil {
|
||||
t.Error("expected error when issue_prefix missing with issues, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetInvariantNames(t *testing.T) {
|
||||
names := GetInvariantNames()
|
||||
|
||||
expectedNames := []string{
|
||||
"foreign_keys_valid",
|
||||
"issue_count_stable",
|
||||
"required_config_present",
|
||||
}
|
||||
|
||||
if len(names) != len(expectedNames) {
|
||||
t.Errorf("expected %d invariants, got %d", len(expectedNames), len(names))
|
||||
}
|
||||
|
||||
for i, name := range names {
|
||||
if name != expectedNames[i] {
|
||||
t.Errorf("expected invariant[%d]=%s, got %s", i, expectedNames[i], name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// setupInvariantTestDB creates an in-memory test database with schema
|
||||
func setupInvariantTestDB(t *testing.T) *sql.DB {
|
||||
t.Helper()
|
||||
|
||||
store := newTestStore(t, ":memory:")
|
||||
t.Cleanup(func() { _ = store.Close() })
|
||||
|
||||
// Return the underlying database connection
|
||||
return store.db
|
||||
}
|
||||
@@ -29,13 +29,66 @@ var migrations = []Migration{
|
||||
{"content_hash_column", migrateContentHashColumn},
|
||||
}
|
||||
|
||||
// RunMigrations executes all registered migrations in order
|
||||
// MigrationInfo contains metadata about a migration for inspection
|
||||
type MigrationInfo struct {
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description"`
|
||||
}
|
||||
|
||||
// ListMigrations returns list of all registered migrations with descriptions
|
||||
// Note: This returns ALL registered migrations, not just pending ones (all are idempotent)
|
||||
func ListMigrations() []MigrationInfo {
|
||||
result := make([]MigrationInfo, len(migrations))
|
||||
for i, m := range migrations {
|
||||
result[i] = MigrationInfo{
|
||||
Name: m.Name,
|
||||
Description: getMigrationDescription(m.Name),
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// getMigrationDescription returns a human-readable description for a migration
|
||||
func getMigrationDescription(name string) string {
|
||||
descriptions := map[string]string{
|
||||
"dirty_issues_table": "Adds dirty_issues table for auto-export tracking",
|
||||
"external_ref_column": "Adds external_ref column to issues table",
|
||||
"composite_indexes": "Adds composite indexes for better query performance",
|
||||
"closed_at_constraint": "Adds constraint ensuring closed issues have closed_at timestamp",
|
||||
"compaction_columns": "Adds compaction tracking columns (compacted_at, compacted_at_commit)",
|
||||
"snapshots_table": "Adds snapshots table for issue history",
|
||||
"compaction_config": "Adds config entries for compaction",
|
||||
"compacted_at_commit_column": "Adds compacted_at_commit to snapshots table",
|
||||
"export_hashes_table": "Adds export_hashes table for idempotent exports",
|
||||
"content_hash_column": "Adds content_hash column for collision resolution",
|
||||
}
|
||||
|
||||
if desc, ok := descriptions[name]; ok {
|
||||
return desc
|
||||
}
|
||||
return "Unknown migration"
|
||||
}
|
||||
|
||||
// RunMigrations executes all registered migrations in order with invariant checking
|
||||
func RunMigrations(db *sql.DB) error {
|
||||
// Capture pre-migration snapshot for validation
|
||||
snapshot, err := captureSnapshot(db)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to capture pre-migration snapshot: %w", err)
|
||||
}
|
||||
|
||||
// Run migrations (they are already idempotent)
|
||||
for _, migration := range migrations {
|
||||
if err := migration.Func(db); err != nil {
|
||||
return fmt.Errorf("migration %s failed: %w", migration.Name, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Verify invariants after migrations complete
|
||||
if err := verifyInvariants(db, snapshot); err != nil {
|
||||
return fmt.Errorf("post-migration validation failed: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user