Remove compaction planning docs (issues filed as bd-251 to bd-267)
This commit is contained in:
1654
COMPACTION_DESIGN.md
1654
COMPACTION_DESIGN.md
File diff suppressed because it is too large
Load Diff
@@ -1,285 +0,0 @@
|
|||||||
# Issue Compaction - Quick Reference
|
|
||||||
|
|
||||||
**Status:** Design Complete, Ready for Implementation
|
|
||||||
**Target:** Beads v1.1
|
|
||||||
**Estimated Effort:** 40-60 hours (MVP)
|
|
||||||
|
|
||||||
## What Is This?
|
|
||||||
|
|
||||||
Add intelligent database compaction to beads using Claude Haiku to semantically compress old, closed issues. Keep your database lightweight while preserving the meaning of past work.
|
|
||||||
|
|
||||||
## Why?
|
|
||||||
|
|
||||||
- **Agent Efficiency:** Smaller DBs → faster queries → clearer thinking
|
|
||||||
- **Context Management:** Agents need summaries, not verbose details
|
|
||||||
- **Pragmatic:** Most work is throwaway; forensic value decays exponentially
|
|
||||||
- **Git-Friendly:** Smaller JSONL exports
|
|
||||||
|
|
||||||
## How It Works
|
|
||||||
|
|
||||||
### Two-Tier Compression
|
|
||||||
|
|
||||||
**Tier 1: Standard (30 days)**
|
|
||||||
- Closed ≥30 days, all dependents closed (2 levels deep)
|
|
||||||
- Haiku summarizes to ~300 words
|
|
||||||
- Keeps: title, summary with key decisions
|
|
||||||
- Clears: verbose design/notes/criteria
|
|
||||||
- **Result:** 70-85% space reduction
|
|
||||||
|
|
||||||
**Tier 2: Aggressive (90 days)**
|
|
||||||
- Already Tier 1 compressed
|
|
||||||
- Closed ≥90 days, deep dependencies closed (5 levels)
|
|
||||||
- ≥100 git commits since closure (measures "project time")
|
|
||||||
- Ultra-compress to single ≤150 word paragraph
|
|
||||||
- Optionally prunes old events
|
|
||||||
- **Result:** 90-95% space reduction
|
|
||||||
|
|
||||||
### Safety First
|
|
||||||
|
|
||||||
- **Snapshots:** Full original content saved before compaction
|
|
||||||
- **Restore:** `bd compact --restore <id>` undoes compaction
|
|
||||||
- **Dry-Run:** `bd compact --dry-run` previews without changing anything
|
|
||||||
- **Git Backup:** JSONL commits preserve full history
|
|
||||||
- **Opt-In:** Disabled by default (`auto_compact_enabled = false`)
|
|
||||||
|
|
||||||
## CLI
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Preview candidates
|
|
||||||
bd compact --dry-run
|
|
||||||
|
|
||||||
# Compact Tier 1
|
|
||||||
bd compact --all
|
|
||||||
|
|
||||||
# Compact Tier 2
|
|
||||||
bd compact --tier 2 --all
|
|
||||||
|
|
||||||
# Compact specific issue
|
|
||||||
bd compact --id bd-42
|
|
||||||
|
|
||||||
# Restore compacted issue
|
|
||||||
bd compact --restore bd-42
|
|
||||||
|
|
||||||
# Show statistics
|
|
||||||
bd compact --stats
|
|
||||||
```
|
|
||||||
|
|
||||||
## Cost
|
|
||||||
|
|
||||||
**Haiku Pricing:**
|
|
||||||
- ~$0.0008 per issue (Tier 1)
|
|
||||||
- ~$0.0003 per issue (Tier 2)
|
|
||||||
- **~$1.10 per 1,000 issues total**
|
|
||||||
|
|
||||||
Negligible for typical usage (~$0.05/month for active project).
|
|
||||||
|
|
||||||
## Example Output
|
|
||||||
|
|
||||||
### Before Compaction
|
|
||||||
```
|
|
||||||
bd-42: Fix authentication bug [CLOSED]
|
|
||||||
|
|
||||||
Description: (800 words about the problem)
|
|
||||||
|
|
||||||
Design: (1,200 words of implementation notes)
|
|
||||||
|
|
||||||
Notes: (500 words of testing/deployment details)
|
|
||||||
|
|
||||||
Acceptance Criteria: (300 words of requirements)
|
|
||||||
|
|
||||||
Total: 2,341 bytes
|
|
||||||
```
|
|
||||||
|
|
||||||
### After Tier 1 Compaction
|
|
||||||
```
|
|
||||||
bd-42: Fix authentication bug [CLOSED] 🗜️
|
|
||||||
|
|
||||||
**Summary:** Fixed race condition in JWT token refresh logic causing
|
|
||||||
intermittent 401 errors. Implemented mutex-based locking. All users
|
|
||||||
can now stay authenticated reliably.
|
|
||||||
|
|
||||||
**Key Decisions:**
|
|
||||||
- Used sync.RWMutex for simpler reasoning
|
|
||||||
- Added exponential backoff to prevent thundering herd
|
|
||||||
- Preserved token format for backward compatibility
|
|
||||||
|
|
||||||
**Resolution:** Deployed Aug 31, zero 401s after 2 weeks monitoring.
|
|
||||||
|
|
||||||
---
|
|
||||||
💾 Restore: bd compact --restore bd-42
|
|
||||||
📊 Original: 2,341 bytes | Compressed: 468 bytes (80% reduction)
|
|
||||||
|
|
||||||
Total: 468 bytes (80% reduction)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Implementation Plan
|
|
||||||
|
|
||||||
### Phase 1: Foundation (16 hours)
|
|
||||||
1. Schema changes (compaction columns, snapshots table)
|
|
||||||
2. Config keys
|
|
||||||
3. Candidate identification queries
|
|
||||||
4. Migration testing
|
|
||||||
|
|
||||||
### Phase 2: Core Compaction (20 hours)
|
|
||||||
5. Haiku client with prompts
|
|
||||||
6. Snapshot creation/restoration
|
|
||||||
7. Tier 1 compaction logic
|
|
||||||
8. CLI command (`bd compact`)
|
|
||||||
|
|
||||||
### Phase 3: Advanced Features (12 hours)
|
|
||||||
9. Tier 2 compaction
|
|
||||||
10. Restore functionality
|
|
||||||
11. Stats command
|
|
||||||
12. Event tracking
|
|
||||||
|
|
||||||
### Phase 4: Polish (12 hours)
|
|
||||||
13. Comprehensive tests
|
|
||||||
14. Documentation (README, COMPACTION.md)
|
|
||||||
15. Examples and workflows
|
|
||||||
|
|
||||||
**Total MVP:** ~60 hours
|
|
||||||
|
|
||||||
### Optional Enhancements (Phase 5+)
|
|
||||||
- Auto-compaction triggers
|
|
||||||
- Git commit counting
|
|
||||||
- Local model support (Ollama)
|
|
||||||
- Custom prompt templates
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
```
|
|
||||||
Issue (closed 45 days ago)
|
|
||||||
↓
|
|
||||||
Eligibility Check
|
|
||||||
↓ (passes: all deps closed, >30 days)
|
|
||||||
Create Snapshot
|
|
||||||
↓
|
|
||||||
Call Haiku API
|
|
||||||
↓ (returns: structured summary)
|
|
||||||
Update Issue
|
|
||||||
↓
|
|
||||||
Record Event
|
|
||||||
↓
|
|
||||||
Export to JSONL
|
|
||||||
```
|
|
||||||
|
|
||||||
## Key Files
|
|
||||||
|
|
||||||
**New Files:**
|
|
||||||
- `internal/compact/haiku.go` - Haiku API client
|
|
||||||
- `internal/compact/compactor.go` - Core compaction logic
|
|
||||||
- `internal/storage/sqlite/compact.go` - Candidate queries
|
|
||||||
- `cmd/bd/compact.go` - CLI command
|
|
||||||
|
|
||||||
**Modified Files:**
|
|
||||||
- `internal/storage/sqlite/schema.go` - Add snapshots table
|
|
||||||
- `internal/storage/sqlite/sqlite.go` - Add migrations
|
|
||||||
- `internal/types/types.go` - Add EventCompacted
|
|
||||||
- `cmd/bd/show.go` - Display compaction status
|
|
||||||
|
|
||||||
## Schema Changes
|
|
||||||
|
|
||||||
```sql
|
|
||||||
-- Add to issues table
|
|
||||||
ALTER TABLE issues ADD COLUMN compaction_level INTEGER DEFAULT 0;
|
|
||||||
ALTER TABLE issues ADD COLUMN compacted_at DATETIME;
|
|
||||||
ALTER TABLE issues ADD COLUMN original_size INTEGER;
|
|
||||||
|
|
||||||
-- New table
|
|
||||||
CREATE TABLE issue_snapshots (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
issue_id TEXT NOT NULL,
|
|
||||||
snapshot_time DATETIME NOT NULL,
|
|
||||||
compaction_level INTEGER NOT NULL,
|
|
||||||
original_size INTEGER NOT NULL,
|
|
||||||
compressed_size INTEGER NOT NULL,
|
|
||||||
original_content TEXT NOT NULL, -- JSON
|
|
||||||
archived_events TEXT,
|
|
||||||
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
```sql
|
|
||||||
INSERT INTO config (key, value) VALUES
|
|
||||||
('compact_tier1_days', '30'),
|
|
||||||
('compact_tier1_dep_levels', '2'),
|
|
||||||
('compact_tier2_days', '90'),
|
|
||||||
('compact_tier2_dep_levels', '5'),
|
|
||||||
('compact_tier2_commits', '100'),
|
|
||||||
('compact_model', 'claude-3-5-haiku-20241022'),
|
|
||||||
('compact_batch_size', '50'),
|
|
||||||
('compact_parallel_workers', '5'),
|
|
||||||
('auto_compact_enabled', 'false');
|
|
||||||
```
|
|
||||||
|
|
||||||
## Haiku Prompts
|
|
||||||
|
|
||||||
### Tier 1 (300 words)
|
|
||||||
```
|
|
||||||
Summarize this closed software issue. Preserve key decisions,
|
|
||||||
implementation approach, and outcome. Max 300 words.
|
|
||||||
|
|
||||||
Title: {{.Title}}
|
|
||||||
Type: {{.IssueType}}
|
|
||||||
Priority: {{.Priority}}
|
|
||||||
|
|
||||||
Description: {{.Description}}
|
|
||||||
Design Notes: {{.Design}}
|
|
||||||
Implementation Notes: {{.Notes}}
|
|
||||||
Acceptance Criteria: {{.AcceptanceCriteria}}
|
|
||||||
|
|
||||||
Output format:
|
|
||||||
**Summary:** [2-3 sentences: problem, solution, outcome]
|
|
||||||
**Key Decisions:** [bullet points of non-obvious choices]
|
|
||||||
**Resolution:** [how it was closed]
|
|
||||||
```
|
|
||||||
|
|
||||||
### Tier 2 (150 words)
|
|
||||||
```
|
|
||||||
Ultra-compress this old closed issue to ≤150 words.
|
|
||||||
Focus on lasting architectural impact.
|
|
||||||
|
|
||||||
Title: {{.Title}}
|
|
||||||
Original Summary: {{.Description}}
|
|
||||||
|
|
||||||
Output a single paragraph covering:
|
|
||||||
- What was built/fixed
|
|
||||||
- Why it mattered
|
|
||||||
- Lasting impact (if any)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Success Metrics
|
|
||||||
|
|
||||||
- **Space:** 70-85% reduction (Tier 1), 90-95% (Tier 2)
|
|
||||||
- **Quality:** Summaries preserve essential context
|
|
||||||
- **Safety:** 100% restore success rate
|
|
||||||
- **Performance:** <100ms candidate query, ~2s per issue (Haiku)
|
|
||||||
- **Cost:** <$1.50 per 1,000 issues
|
|
||||||
|
|
||||||
## Risks & Mitigations
|
|
||||||
|
|
||||||
| Risk | Mitigation |
|
|
||||||
|------|------------|
|
|
||||||
| Info loss in summaries | Dry-run review, restore capability, prompt tuning |
|
|
||||||
| API rate limits | Exponential backoff, respect limits, batch sizing |
|
|
||||||
| Accidental compression | Dry-run required, snapshots permanent |
|
|
||||||
| JSONL conflicts | Compaction is deterministic, git handles well |
|
|
||||||
|
|
||||||
## Next Steps
|
|
||||||
|
|
||||||
1. **Review design:** `COMPACTION_DESIGN.md` (comprehensive spec)
|
|
||||||
2. **File issues:** `bd create -f compaction-issues.md` (16 issues)
|
|
||||||
3. **Start implementation:** Begin with schema + config (Issues 1-2)
|
|
||||||
4. **Iterate:** Build MVP (Tier 1 only), then enhance
|
|
||||||
|
|
||||||
## Questions?
|
|
||||||
|
|
||||||
- Full design: `COMPACTION_DESIGN.md`
|
|
||||||
- Issue breakdown: `compaction-issues.md`
|
|
||||||
- Reference implementation in Phase 1-4 above
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Ready to implement in another session!**
|
|
||||||
@@ -1,779 +0,0 @@
|
|||||||
# Compaction Feature Issues
|
|
||||||
|
|
||||||
This file contains all issues for the database compaction feature, ready to import with:
|
|
||||||
```bash
|
|
||||||
bd create -f compaction-issues.md
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Epic: Add intelligent database compaction with Claude Haiku
|
|
||||||
|
|
||||||
### Type
|
|
||||||
epic
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
2
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Implement multi-tier database compaction using Claude Haiku to semantically compress old, closed issues. This keeps the database lightweight and agent-friendly while preserving essential context.
|
|
||||||
|
|
||||||
Goals:
|
|
||||||
- 70-95% space reduction for eligible issues
|
|
||||||
- Full restore capability via snapshots
|
|
||||||
- Opt-in with dry-run safety
|
|
||||||
- ~$1 per 1,000 issues compacted
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- Schema migration with snapshots table
|
|
||||||
- Haiku integration for summarization
|
|
||||||
- Two-tier compaction (30d, 90d)
|
|
||||||
- CLI with dry-run, restore, stats
|
|
||||||
- Full test coverage
|
|
||||||
- Documentation complete
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, epic, haiku, v1.1
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Add compaction schema and migrations
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
1
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Add database schema support for issue compaction tracking and snapshot storage.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
Add three columns to `issues` table:
|
|
||||||
- `compaction_level INTEGER DEFAULT 0` - 0=original, 1=tier1, 2=tier2
|
|
||||||
- `compacted_at DATETIME` - when last compacted
|
|
||||||
- `original_size INTEGER` - bytes before first compaction
|
|
||||||
|
|
||||||
Create `issue_snapshots` table:
|
|
||||||
```sql
|
|
||||||
CREATE TABLE issue_snapshots (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
issue_id TEXT NOT NULL,
|
|
||||||
snapshot_time DATETIME NOT NULL,
|
|
||||||
compaction_level INTEGER NOT NULL,
|
|
||||||
original_size INTEGER NOT NULL,
|
|
||||||
compressed_size INTEGER NOT NULL,
|
|
||||||
original_content TEXT NOT NULL, -- JSON blob
|
|
||||||
archived_events TEXT,
|
|
||||||
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
Add indexes:
|
|
||||||
- `idx_snapshots_issue` on `issue_id`
|
|
||||||
- `idx_snapshots_level` on `compaction_level`
|
|
||||||
|
|
||||||
Add migration functions in `internal/storage/sqlite/sqlite.go`:
|
|
||||||
- `migrateCompactionColumns(db *sql.DB) error`
|
|
||||||
- `migrateSnapshotsTable(db *sql.DB) error`
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- Existing databases migrate automatically
|
|
||||||
- New databases include columns by default
|
|
||||||
- Migration is idempotent (safe to run multiple times)
|
|
||||||
- No data loss during migration
|
|
||||||
- Tests verify migration on fresh and existing DBs
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, schema, migration, database
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Add compaction configuration keys
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
1
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Add configuration keys for compaction behavior with sensible defaults.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
Add to `internal/storage/sqlite/schema.go` initial config:
|
|
||||||
```sql
|
|
||||||
INSERT OR IGNORE INTO config (key, value) VALUES
|
|
||||||
('compact_tier1_days', '30'),
|
|
||||||
('compact_tier1_dep_levels', '2'),
|
|
||||||
('compact_tier2_days', '90'),
|
|
||||||
('compact_tier2_dep_levels', '5'),
|
|
||||||
('compact_tier2_commits', '100'),
|
|
||||||
('compact_model', 'claude-3-5-haiku-20241022'),
|
|
||||||
('compact_batch_size', '50'),
|
|
||||||
('compact_parallel_workers', '5'),
|
|
||||||
('auto_compact_enabled', 'false');
|
|
||||||
```
|
|
||||||
|
|
||||||
Add helper functions for loading config into typed struct.
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- Config keys created on init
|
|
||||||
- Existing DBs get defaults on migration
|
|
||||||
- `bd config get/set` works with all keys
|
|
||||||
- Type validation (days=int, enabled=bool)
|
|
||||||
- Documentation in README.md
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, config, configuration
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Implement candidate identification queries
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
1
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Write SQL queries to identify issues eligible for Tier 1 and Tier 2 compaction based on closure time and dependency status.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
Create `internal/storage/sqlite/compact.go` with:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type CompactionCandidate struct {
|
|
||||||
IssueID string
|
|
||||||
ClosedAt time.Time
|
|
||||||
OriginalSize int
|
|
||||||
EstimatedSize int
|
|
||||||
DependentCount int
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *SQLiteStorage) GetTier1Candidates(ctx context.Context) ([]*CompactionCandidate, error)
|
|
||||||
func (s *SQLiteStorage) GetTier2Candidates(ctx context.Context) ([]*CompactionCandidate, error)
|
|
||||||
func (s *SQLiteStorage) CheckEligibility(ctx context.Context, issueID string, tier int) (bool, string, error)
|
|
||||||
```
|
|
||||||
|
|
||||||
Use recursive CTE for dependency depth checking (similar to ready_issues view).
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- Tier 1 query filters by days and dependency depth
|
|
||||||
- Tier 2 query includes commit/issue count checks
|
|
||||||
- Dependency checking handles circular deps gracefully
|
|
||||||
- Performance: <100ms for 10,000 issue database
|
|
||||||
- Tests cover edge cases (no deps, circular deps, mixed status)
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, sql, query, dependencies
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Create Haiku client and prompt templates
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
1
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Implement Claude Haiku API client with template-based prompts for Tier 1 and Tier 2 summarization.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
Create `internal/compact/haiku.go`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type HaikuClient struct {
|
|
||||||
client *anthropic.Client
|
|
||||||
model string
|
|
||||||
}
|
|
||||||
|
|
||||||
func NewHaikuClient(apiKey string) (*HaikuClient, error)
|
|
||||||
func (h *HaikuClient) SummarizeTier1(ctx context.Context, issue *types.Issue) (string, error)
|
|
||||||
func (h *HaikuClient) SummarizeTier2(ctx context.Context, issue *types.Issue) (string, error)
|
|
||||||
```
|
|
||||||
|
|
||||||
Use text/template for prompt rendering.
|
|
||||||
|
|
||||||
Tier 1 output format:
|
|
||||||
```
|
|
||||||
**Summary:** [2-3 sentences]
|
|
||||||
**Key Decisions:** [bullet points]
|
|
||||||
**Resolution:** [outcome]
|
|
||||||
```
|
|
||||||
|
|
||||||
Tier 2 output format:
|
|
||||||
```
|
|
||||||
Single paragraph ≤150 words covering what was built, why it mattered, lasting impact.
|
|
||||||
```
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- API key from env var or config (env takes precedence)
|
|
||||||
- Prompts render correctly with templates
|
|
||||||
- Rate limiting handled gracefully (exponential backoff)
|
|
||||||
- Network errors retry up to 3 times
|
|
||||||
- Mock tests for API calls
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, haiku, api, llm
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Implement snapshot creation and restoration
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
1
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Implement snapshot creation before compaction and restoration capability to undo compaction.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
Add to `internal/storage/sqlite/compact.go`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
func (s *SQLiteStorage) CreateSnapshot(ctx context.Context, issue *types.Issue, level int) error
|
|
||||||
func (s *SQLiteStorage) RestoreFromSnapshot(ctx context.Context, issueID string, level int) error
|
|
||||||
func (s *SQLiteStorage) GetSnapshots(ctx context.Context, issueID string) ([]*Snapshot, error)
|
|
||||||
```
|
|
||||||
|
|
||||||
Snapshot JSON structure:
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"description": "...",
|
|
||||||
"design": "...",
|
|
||||||
"notes": "...",
|
|
||||||
"acceptance_criteria": "...",
|
|
||||||
"title": "..."
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- Snapshot created atomically with compaction
|
|
||||||
- Restore returns exact original content
|
|
||||||
- Multiple snapshots per issue supported (Tier 1 → Tier 2)
|
|
||||||
- JSON encoding handles UTF-8 and special characters
|
|
||||||
- Size calculation is accurate (UTF-8 bytes)
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, snapshot, restore, safety
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Implement Tier 1 compaction logic
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
1
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Implement the core Tier 1 compaction process: snapshot → summarize → update.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
Add to `internal/compact/compactor.go`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type Compactor struct {
|
|
||||||
store storage.Storage
|
|
||||||
haiku *HaikuClient
|
|
||||||
config *CompactConfig
|
|
||||||
}
|
|
||||||
|
|
||||||
func New(store storage.Storage, apiKey string, config *CompactConfig) (*Compactor, error)
|
|
||||||
func (c *Compactor) CompactTier1(ctx context.Context, issueID string) error
|
|
||||||
func (c *Compactor) CompactTier1Batch(ctx context.Context, issueIDs []string) error
|
|
||||||
```
|
|
||||||
|
|
||||||
Process:
|
|
||||||
1. Verify eligibility
|
|
||||||
2. Calculate original size
|
|
||||||
3. Create snapshot
|
|
||||||
4. Call Haiku for summary
|
|
||||||
5. Update issue (description=summary, clear design/notes/criteria)
|
|
||||||
6. Set compaction_level=1, compacted_at=now, original_size
|
|
||||||
7. Record EventCompacted
|
|
||||||
8. Mark dirty for export
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- Single issue compaction works end-to-end
|
|
||||||
- Batch processing with parallel workers (5 concurrent)
|
|
||||||
- Errors don't corrupt database (transaction rollback)
|
|
||||||
- EventCompacted includes size savings
|
|
||||||
- Dry-run mode (identify + size estimate only, no API calls)
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, tier1, core-logic
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Implement Tier 2 compaction logic
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
2
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Implement Tier 2 ultra-compression: more aggressive summarization and optional event pruning.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
Add to `internal/compact/compactor.go`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
func (c *Compactor) CompactTier2(ctx context.Context, issueID string) error
|
|
||||||
func (c *Compactor) CompactTier2Batch(ctx context.Context, issueIDs []string) error
|
|
||||||
```
|
|
||||||
|
|
||||||
Process:
|
|
||||||
1. Verify issue is at compaction_level = 1
|
|
||||||
2. Check Tier 2 eligibility (days, deps, commits/issues)
|
|
||||||
3. Create Tier 2 snapshot
|
|
||||||
4. Call Haiku with ultra-compression prompt
|
|
||||||
5. Update issue (description = single paragraph, clear all other fields)
|
|
||||||
6. Set compaction_level = 2
|
|
||||||
7. Optionally prune events (keep created/closed, archive rest to snapshot)
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- Requires existing Tier 1 compaction
|
|
||||||
- Git commit counting works (with fallback to issue counter)
|
|
||||||
- Events optionally pruned (config: compact_events_enabled)
|
|
||||||
- Archived events stored in snapshot JSON
|
|
||||||
- Size reduction 90-95%
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, tier2, advanced
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Add `bd compact` CLI command
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
1
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Implement the `bd compact` command with dry-run, batch processing, and progress reporting.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
Create `cmd/bd/compact.go`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
var compactCmd = &cobra.Command{
|
|
||||||
Use: "compact",
|
|
||||||
Short: "Compact old closed issues to save space",
|
|
||||||
}
|
|
||||||
|
|
||||||
Flags:
|
|
||||||
--dry-run Preview without compacting
|
|
||||||
--tier int Compaction tier (1 or 2, default: 1)
|
|
||||||
--all Process all candidates
|
|
||||||
--id string Compact specific issue
|
|
||||||
--force Force compact (bypass checks, requires --id)
|
|
||||||
--batch-size int Issues per batch
|
|
||||||
--workers int Parallel workers
|
|
||||||
--json JSON output
|
|
||||||
```
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- `--dry-run` shows accurate preview with size estimates
|
|
||||||
- `--all` processes all candidates
|
|
||||||
- `--id` compacts single issue
|
|
||||||
- `--force` bypasses eligibility checks (only with --id)
|
|
||||||
- Progress bar for batches (e.g., [████████] 47/47)
|
|
||||||
- JSON output with `--json`
|
|
||||||
- Exit codes: 0=success, 1=error
|
|
||||||
- Shows summary: count, size saved, cost, time
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, cli, command
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Add `bd compact --restore` functionality
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
2
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Implement restore command to undo compaction from snapshots.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
Add to `cmd/bd/compact.go`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
var compactRestore string
|
|
||||||
|
|
||||||
compactCmd.Flags().StringVar(&compactRestore, "restore", "", "Restore issue from snapshot")
|
|
||||||
```
|
|
||||||
|
|
||||||
Process:
|
|
||||||
1. Load snapshot for issue
|
|
||||||
2. Parse JSON content
|
|
||||||
3. Update issue with original content
|
|
||||||
4. Set compaction_level = 0, compacted_at = NULL, original_size = NULL
|
|
||||||
5. Record event (EventRestored or EventUpdated)
|
|
||||||
6. Mark dirty for export
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- Restores exact original content
|
|
||||||
- Handles multiple snapshots (use latest by default)
|
|
||||||
- `--level` flag to choose specific snapshot
|
|
||||||
- Updates compaction_level correctly
|
|
||||||
- Exports restored content to JSONL
|
|
||||||
- Shows before/after in output
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, restore, cli
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Add `bd compact --stats` command
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
2
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Add statistics command showing compaction status and potential savings.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
```go
|
|
||||||
var compactStats bool
|
|
||||||
|
|
||||||
compactCmd.Flags().BoolVar(&compactStats, "stats", false, "Show compaction statistics")
|
|
||||||
```
|
|
||||||
|
|
||||||
Output:
|
|
||||||
- Total issues, by compaction level (0, 1, 2)
|
|
||||||
- Current DB size vs estimated uncompacted size
|
|
||||||
- Space savings (KB/MB and %)
|
|
||||||
- Candidates for each tier with size estimates
|
|
||||||
- Estimated API cost (Haiku pricing)
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- Accurate counts by compaction_level
|
|
||||||
- Size calculations include all text fields (UTF-8 bytes)
|
|
||||||
- Shows candidates with eligibility reasons
|
|
||||||
- Cost estimation based on current Haiku pricing
|
|
||||||
- JSON output supported
|
|
||||||
- Clear, readable table format
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, stats, reporting
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Add EventCompacted to event system
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
2
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Add new event type for tracking compaction in audit trail.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
1. Add to `internal/types/types.go`:
|
|
||||||
```go
|
|
||||||
const EventCompacted EventType = "compacted"
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Record event during compaction:
|
|
||||||
```go
|
|
||||||
eventData := map[string]interface{}{
|
|
||||||
"tier": tier,
|
|
||||||
"original_size": originalSize,
|
|
||||||
"compressed_size": compressedSize,
|
|
||||||
"reduction_pct": (1 - float64(compressedSize)/float64(originalSize)) * 100,
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Update event display in `bd show`.
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- Event includes tier, original_size, compressed_size, reduction_pct
|
|
||||||
- Shows in event history (`bd events <id>`)
|
|
||||||
- Exports to JSONL correctly
|
|
||||||
- `bd show` displays compaction status and marker
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, events, audit
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Add compaction indicator to `bd show`
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
2
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Update `bd show` command to display compaction status prominently.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
Add to issue display:
|
|
||||||
```
|
|
||||||
bd-42: Fix authentication bug [CLOSED] 🗜️
|
|
||||||
|
|
||||||
Status: closed (compacted L1)
|
|
||||||
...
|
|
||||||
|
|
||||||
---
|
|
||||||
💾 Restore: bd compact --restore bd-42
|
|
||||||
📊 Original: 2,341 bytes | Compressed: 468 bytes (80% reduction)
|
|
||||||
🗜️ Compacted: 2025-10-15 (Tier 1)
|
|
||||||
```
|
|
||||||
|
|
||||||
Emoji indicators:
|
|
||||||
- Tier 1: 🗜️
|
|
||||||
- Tier 2: 📦
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- Compaction status visible in title line
|
|
||||||
- Footer shows size savings when compacted
|
|
||||||
- Restore command shown for compacted issues
|
|
||||||
- Works with `--json` output (includes compaction fields)
|
|
||||||
- Emoji optional (controlled by config or terminal detection)
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, ui, display
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Write compaction tests
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
1
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Comprehensive test suite for compaction functionality.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
Test coverage:
|
|
||||||
|
|
||||||
1. **Candidate Identification:**
|
|
||||||
- Eligibility by time
|
|
||||||
- Dependency depth checking
|
|
||||||
- Mixed status dependents
|
|
||||||
- Edge cases (no deps, circular deps)
|
|
||||||
|
|
||||||
2. **Snapshots:**
|
|
||||||
- Create and restore
|
|
||||||
- Multiple snapshots per issue
|
|
||||||
- Content integrity (UTF-8, special chars)
|
|
||||||
|
|
||||||
3. **Tier 1 Compaction:**
|
|
||||||
- Single issue compaction
|
|
||||||
- Batch processing
|
|
||||||
- Error handling (API failures)
|
|
||||||
|
|
||||||
4. **Tier 2 Compaction:**
|
|
||||||
- Requires Tier 1
|
|
||||||
- Events pruning
|
|
||||||
- Commit counting fallback
|
|
||||||
|
|
||||||
5. **CLI:**
|
|
||||||
- All flag combinations
|
|
||||||
- Dry-run accuracy
|
|
||||||
- JSON output parsing
|
|
||||||
|
|
||||||
6. **Integration:**
|
|
||||||
- End-to-end flow
|
|
||||||
- JSONL export/import
|
|
||||||
- Restore verification
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- Test coverage >80%
|
|
||||||
- All edge cases covered
|
|
||||||
- Mock Haiku API in tests (no real API calls)
|
|
||||||
- Integration tests pass
|
|
||||||
- `go test ./...` passes
|
|
||||||
- Benchmarks for performance-critical paths
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, testing, quality
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Add compaction documentation
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
2
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Document compaction feature in README and create detailed COMPACTION.md guide.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
**Update README.md:**
|
|
||||||
- Add to Features section
|
|
||||||
- CLI examples (dry-run, compact, restore, stats)
|
|
||||||
- Configuration guide
|
|
||||||
- Cost analysis
|
|
||||||
|
|
||||||
**Create COMPACTION.md:**
|
|
||||||
- How compaction works (architecture overview)
|
|
||||||
- When to use each tier
|
|
||||||
- Detailed cost analysis with examples
|
|
||||||
- Safety mechanisms (snapshots, restore, dry-run)
|
|
||||||
- Troubleshooting guide
|
|
||||||
- FAQ
|
|
||||||
|
|
||||||
**Create examples/compaction/:**
|
|
||||||
- `workflow.sh` - Example monthly compaction workflow
|
|
||||||
- `cron-compact.sh` - Cron job setup
|
|
||||||
- `auto-compact.sh` - Auto-compaction script
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- README.md updated with compaction section
|
|
||||||
- COMPACTION.md comprehensive and clear
|
|
||||||
- Examples work as documented (tested)
|
|
||||||
- Screenshots or ASCII examples included
|
|
||||||
- API key setup documented (env var vs config)
|
|
||||||
- Covers common questions and issues
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, docs, documentation, examples
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Optional: Implement auto-compaction
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
3
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Implement automatic compaction triggered by certain operations when enabled via config.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
Trigger points (when `auto_compact_enabled = true`):
|
|
||||||
1. `bd stats` - check and compact if candidates exist
|
|
||||||
2. `bd export` - before exporting
|
|
||||||
3. Configurable: on any read operation after N candidates accumulate
|
|
||||||
|
|
||||||
Add:
|
|
||||||
```go
|
|
||||||
func (s *SQLiteStorage) AutoCompact(ctx context.Context) error {
|
|
||||||
enabled, _ := s.GetConfig(ctx, "auto_compact_enabled")
|
|
||||||
if enabled != "true" {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// Run Tier 1 compaction on all candidates
|
|
||||||
// Limit to batch_size to avoid long operations
|
|
||||||
// Log activity for transparency
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- Respects auto_compact_enabled config (default: false)
|
|
||||||
- Limits batch size to avoid blocking operations
|
|
||||||
- Logs compaction activity (visible with --verbose)
|
|
||||||
- Can be disabled per-command with `--no-auto-compact` flag
|
|
||||||
- Only compacts Tier 1 (Tier 2 remains manual)
|
|
||||||
- Doesn't run more than once per hour (rate limiting)
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, automation, optional, v1.2
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Optional: Add git commit counting
|
|
||||||
|
|
||||||
### Type
|
|
||||||
task
|
|
||||||
|
|
||||||
### Priority
|
|
||||||
3
|
|
||||||
|
|
||||||
### Description
|
|
||||||
|
|
||||||
Implement git commit counting for "project time" measurement as alternative to calendar time for Tier 2 eligibility.
|
|
||||||
|
|
||||||
### Design
|
|
||||||
|
|
||||||
```go
|
|
||||||
func getCommitsSince(closedAt time.Time) (int, error) {
|
|
||||||
cmd := exec.Command("git", "rev-list", "--count",
|
|
||||||
fmt.Sprintf("--since=%s", closedAt.Format(time.RFC3339)), "HEAD")
|
|
||||||
output, err := cmd.Output()
|
|
||||||
if err != nil {
|
|
||||||
return 0, err // Not in git repo or git not available
|
|
||||||
}
|
|
||||||
return strconv.Atoi(strings.TrimSpace(string(output)))
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Fallback strategies:
|
|
||||||
1. Git commit count (preferred)
|
|
||||||
2. Issue counter delta (store counter at close time, compare later)
|
|
||||||
3. Pure time-based (90 days)
|
|
||||||
|
|
||||||
### Acceptance Criteria
|
|
||||||
- Counts commits since closed_at timestamp
|
|
||||||
- Handles git not available gracefully (falls back)
|
|
||||||
- Fallback to issue counter delta works
|
|
||||||
- Configurable via compact_tier2_commits config key
|
|
||||||
- Tested with real git repo
|
|
||||||
- Works in non-git environments
|
|
||||||
|
|
||||||
### Labels
|
|
||||||
compaction, git, optional, tier2
|
|
||||||
Reference in New Issue
Block a user