Add comprehensive compaction documentation
- Updated README.md with Tier 1/2 info, restore command, cost analysis - Created COMPACTION.md with full guide covering: - How compaction works (architecture, two-tier system) - CLI reference and examples - Eligibility rules and configuration - Cost analysis with detailed tables - Automation examples (cron, workflows) - Safety, recovery, and troubleshooting - FAQ and best practices - Added examples/compaction/ with 3 scripts: - workflow.sh: Interactive compaction workflow - cron-compact.sh: Automated monthly compaction - auto-compact.sh: Smart threshold-based compaction - README.md: Examples documentation Closes bd-265 Amp-Thread-ID: https://ampcode.com/threads/T-8113e88e-1cd0-4a9e-b581-07045a3ed31e Co-authored-by: Amp <amp@ampcode.com>
This commit is contained in:
Steve Yegge
186
examples/compaction/README.md
Normal file
186
examples/compaction/README.md
Normal file
@@ -0,0 +1,186 @@
|
||||
# Compaction Examples
|
||||
|
||||
This directory contains example scripts for automating database compaction.
|
||||
|
||||
## Scripts
|
||||
|
||||
### workflow.sh
|
||||
|
||||
Interactive compaction workflow with prompts. Perfect for manual compaction runs.
|
||||
|
||||
```bash
|
||||
chmod +x workflow.sh
|
||||
export ANTHROPIC_API_KEY="sk-ant-..."
|
||||
./workflow.sh
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Previews candidates before compaction
|
||||
- Prompts for confirmation at each tier
|
||||
- Shows final statistics
|
||||
- Provides next-step guidance
|
||||
|
||||
**When to use:** Manual monthly/quarterly compaction
|
||||
|
||||
### cron-compact.sh
|
||||
|
||||
Fully automated compaction for cron jobs. No interaction required.
|
||||
|
||||
```bash
|
||||
# Configure
|
||||
export BD_REPO_PATH="/path/to/your/repo"
|
||||
export BD_LOG_FILE="$HOME/.bd-compact.log"
|
||||
export ANTHROPIC_API_KEY="sk-ant-..."
|
||||
|
||||
# Test manually
|
||||
./cron-compact.sh
|
||||
|
||||
# Install to cron (monthly)
|
||||
cp cron-compact.sh /etc/cron.monthly/bd-compact
|
||||
chmod +x /etc/cron.monthly/bd-compact
|
||||
|
||||
# Or add to crontab
|
||||
crontab -e
|
||||
# Add: 0 2 1 * * /path/to/cron-compact.sh
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Pulls latest changes before compacting
|
||||
- Logs all output
|
||||
- Auto-commits and pushes results
|
||||
- Reports counts of compacted issues
|
||||
|
||||
**When to use:** Automated monthly compaction for active projects
|
||||
|
||||
### auto-compact.sh
|
||||
|
||||
Smart auto-compaction with thresholds. Only runs if enough eligible issues exist.
|
||||
|
||||
```bash
|
||||
chmod +x auto-compact.sh
|
||||
|
||||
# Compact if 10+ eligible issues
|
||||
./auto-compact.sh
|
||||
|
||||
# Custom threshold
|
||||
./auto-compact.sh --threshold 50
|
||||
|
||||
# Tier 2 ultra-compression
|
||||
./auto-compact.sh --tier 2 --threshold 20
|
||||
|
||||
# Preview without compacting
|
||||
./auto-compact.sh --dry-run
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Configurable eligibility threshold
|
||||
- Skips compaction if below threshold
|
||||
- Supports both tiers
|
||||
- Dry-run mode for testing
|
||||
|
||||
**When to use:**
|
||||
- Pre-commit hooks (if ANTHROPIC_API_KEY set)
|
||||
- CI/CD pipelines
|
||||
- Conditional automation
|
||||
|
||||
## Configuration
|
||||
|
||||
All scripts require:
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_API_KEY="sk-ant-..."
|
||||
```
|
||||
|
||||
Additional environment variables:
|
||||
|
||||
- `BD_REPO_PATH`: Repository path (cron-compact.sh)
|
||||
- `BD_LOG_FILE`: Log file location (cron-compact.sh)
|
||||
|
||||
## Recommendations
|
||||
|
||||
### Small Projects (<500 issues)
|
||||
Use `workflow.sh` manually, once or twice per year.
|
||||
|
||||
### Medium Projects (500-5000 issues)
|
||||
Use `cron-compact.sh` quarterly or `auto-compact.sh` in CI.
|
||||
|
||||
### Large Projects (5000+ issues)
|
||||
Use `cron-compact.sh` monthly with both tiers:
|
||||
```bash
|
||||
# Modify cron-compact.sh to run both tiers
|
||||
```
|
||||
|
||||
### High-Velocity Teams
|
||||
Combine approaches:
|
||||
- `auto-compact.sh --threshold 50` in CI (Tier 1 only)
|
||||
- `cron-compact.sh` monthly for Tier 2
|
||||
|
||||
## Testing
|
||||
|
||||
Before deploying to cron, test scripts manually:
|
||||
|
||||
```bash
|
||||
# Test workflow
|
||||
export ANTHROPIC_API_KEY="sk-ant-..."
|
||||
./workflow.sh
|
||||
|
||||
# Test cron script
|
||||
export BD_REPO_PATH="$(pwd)"
|
||||
./cron-compact.sh
|
||||
|
||||
# Test auto-compact (dry run)
|
||||
./auto-compact.sh --dry-run --threshold 1
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Script says "bd command not found"
|
||||
|
||||
Ensure bd is in PATH:
|
||||
```bash
|
||||
which bd
|
||||
export PATH="$PATH:/usr/local/bin"
|
||||
```
|
||||
|
||||
### "ANTHROPIC_API_KEY not set"
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_API_KEY="sk-ant-..."
|
||||
# Add to ~/.zshrc or ~/.bashrc for persistence
|
||||
```
|
||||
|
||||
### Cron job not running
|
||||
|
||||
Check cron logs:
|
||||
```bash
|
||||
# Linux
|
||||
grep CRON /var/log/syslog
|
||||
|
||||
# macOS
|
||||
log show --predicate 'process == "cron"' --last 1h
|
||||
```
|
||||
|
||||
Verify script is executable:
|
||||
```bash
|
||||
chmod +x /etc/cron.monthly/bd-compact
|
||||
```
|
||||
|
||||
## Cost Monitoring
|
||||
|
||||
Track compaction costs:
|
||||
|
||||
```bash
|
||||
# Show stats after compaction
|
||||
bd compact --stats
|
||||
|
||||
# Estimate monthly cost
|
||||
# (issues_compacted / 1000) * $1.00
|
||||
```
|
||||
|
||||
Set up alerts if costs exceed budget (future feature: bd-cost-alert).
|
||||
|
||||
## See Also
|
||||
|
||||
- [COMPACTION.md](../../COMPACTION.md) - Comprehensive compaction guide
|
||||
- [README.md](../../README.md) - Main documentation
|
||||
- [GIT_WORKFLOW.md](../../GIT_WORKFLOW.md) - Multi-machine collaboration
|
||||
79
examples/compaction/auto-compact.sh
Executable file
79
examples/compaction/auto-compact.sh
Executable file
@@ -0,0 +1,79 @@
|
||||
#!/bin/bash
|
||||
# Smart auto-compaction with thresholds
|
||||
# Only compacts if there are enough eligible issues
|
||||
#
|
||||
# Usage: ./auto-compact.sh [--threshold N] [--tier 1|2]
|
||||
|
||||
# Default configuration
|
||||
THRESHOLD=10 # Minimum eligible issues to trigger compaction
|
||||
TIER=1
|
||||
DRY_RUN=false
|
||||
|
||||
# Parse arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--threshold)
|
||||
THRESHOLD="$2"
|
||||
shift 2
|
||||
;;
|
||||
--tier)
|
||||
TIER="$2"
|
||||
shift 2
|
||||
;;
|
||||
--dry-run)
|
||||
DRY_RUN=true
|
||||
shift
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1"
|
||||
echo "Usage: $0 [--threshold N] [--tier 1|2] [--dry-run]"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Check API key
|
||||
if [ -z "$ANTHROPIC_API_KEY" ]; then
|
||||
echo "❌ Error: ANTHROPIC_API_KEY not set"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check bd is installed
|
||||
if ! command -v bd &> /dev/null; then
|
||||
echo "❌ Error: bd command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check eligible issues
|
||||
echo "Checking eligible issues (Tier $TIER)..."
|
||||
ELIGIBLE=$(bd compact --dry-run --all --tier "$TIER" --json 2>/dev/null | jq '. | length' || echo "0")
|
||||
|
||||
if [ -z "$ELIGIBLE" ] || [ "$ELIGIBLE" = "null" ]; then
|
||||
ELIGIBLE=0
|
||||
fi
|
||||
|
||||
echo "Found $ELIGIBLE eligible issues (threshold: $THRESHOLD)"
|
||||
|
||||
if [ "$ELIGIBLE" -lt "$THRESHOLD" ]; then
|
||||
echo "⏭️ Below threshold, skipping compaction"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$DRY_RUN" = true ]; then
|
||||
echo "🔍 Dry run mode - showing candidates:"
|
||||
bd compact --dry-run --all --tier "$TIER"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Run compaction
|
||||
echo "🗜️ Compacting $ELIGIBLE issues (Tier $TIER)..."
|
||||
bd compact --all --tier "$TIER"
|
||||
|
||||
# Show stats
|
||||
echo
|
||||
echo "📊 Statistics:"
|
||||
bd compact --stats
|
||||
|
||||
echo
|
||||
echo "✅ Auto-compaction complete"
|
||||
echo "Remember to commit: git add .beads/issues.jsonl issues.db && git commit -m 'Auto-compact'"
|
||||
75
examples/compaction/cron-compact.sh
Executable file
75
examples/compaction/cron-compact.sh
Executable file
@@ -0,0 +1,75 @@
|
||||
#!/bin/bash
|
||||
# Automated monthly compaction for cron
|
||||
# Install: cp cron-compact.sh /etc/cron.monthly/bd-compact
|
||||
# chmod +x /etc/cron.monthly/bd-compact
|
||||
#
|
||||
# Or add to crontab:
|
||||
# 0 2 1 * * /path/to/cron-compact.sh
|
||||
|
||||
# Configuration
|
||||
REPO_PATH="${BD_REPO_PATH:-$HOME/your-project}"
|
||||
LOG_FILE="${BD_LOG_FILE:-$HOME/.bd-compact.log}"
|
||||
API_KEY="${ANTHROPIC_API_KEY}"
|
||||
|
||||
# Exit on error
|
||||
set -e
|
||||
|
||||
# Logging helper
|
||||
log() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
|
||||
}
|
||||
|
||||
log "=== Starting BD Compaction ==="
|
||||
|
||||
# Check API key
|
||||
if [ -z "$API_KEY" ]; then
|
||||
log "ERROR: ANTHROPIC_API_KEY not set"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Change to repo directory
|
||||
if [ ! -d "$REPO_PATH" ]; then
|
||||
log "ERROR: Repository not found: $REPO_PATH"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cd "$REPO_PATH"
|
||||
log "Repository: $(pwd)"
|
||||
|
||||
# Check bd is installed
|
||||
if ! command -v bd &> /dev/null; then
|
||||
log "ERROR: bd command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Pull latest changes
|
||||
log "Pulling latest changes..."
|
||||
git pull origin main 2>&1 | tee -a "$LOG_FILE"
|
||||
|
||||
# Tier 1 compaction
|
||||
log "Running Tier 1 compaction..."
|
||||
TIER1_COUNT=$(bd compact --all --json 2>&1 | jq '. | length' || echo "0")
|
||||
log "Compacted $TIER1_COUNT Tier 1 issues"
|
||||
|
||||
# Tier 2 compaction
|
||||
log "Running Tier 2 compaction..."
|
||||
TIER2_COUNT=$(bd compact --all --tier 2 --json 2>&1 | jq '. | length' || echo "0")
|
||||
log "Compacted $TIER2_COUNT Tier 2 issues"
|
||||
|
||||
# Show statistics
|
||||
log "Compaction statistics:"
|
||||
bd compact --stats 2>&1 | tee -a "$LOG_FILE"
|
||||
|
||||
# Commit and push if changes exist
|
||||
if git diff --quiet .beads/issues.jsonl issues.db 2>/dev/null; then
|
||||
log "No changes to commit"
|
||||
else
|
||||
log "Committing compaction results..."
|
||||
git add .beads/issues.jsonl issues.db
|
||||
git commit -m "Automated compaction: $(date +%Y-%m-%d) - T1:$TIER1_COUNT T2:$TIER2_COUNT"
|
||||
git push origin main 2>&1 | tee -a "$LOG_FILE"
|
||||
log "Changes pushed to remote"
|
||||
fi
|
||||
|
||||
log "=== Compaction Complete ==="
|
||||
log "Total compacted: $((TIER1_COUNT + TIER2_COUNT)) issues"
|
||||
73
examples/compaction/workflow.sh
Executable file
73
examples/compaction/workflow.sh
Executable file
@@ -0,0 +1,73 @@
|
||||
#!/bin/bash
|
||||
# Interactive compaction workflow
|
||||
# Run this manually when you want to compact old issues
|
||||
|
||||
set -e
|
||||
|
||||
echo "=== BD Compaction Workflow ==="
|
||||
echo "Date: $(date)"
|
||||
echo
|
||||
|
||||
# Check API key
|
||||
if [ -z "$ANTHROPIC_API_KEY" ]; then
|
||||
echo "❌ Error: ANTHROPIC_API_KEY not set"
|
||||
echo
|
||||
echo "Set your API key:"
|
||||
echo " export ANTHROPIC_API_KEY='sk-ant-...'"
|
||||
echo
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check bd is installed
|
||||
if ! command -v bd &> /dev/null; then
|
||||
echo "❌ Error: bd command not found"
|
||||
echo
|
||||
echo "Install bd:"
|
||||
echo " curl -fsSL https://raw.githubusercontent.com/steveyegge/beads/main/install.sh | bash"
|
||||
echo
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Preview candidates
|
||||
echo "--- Preview Tier 1 Candidates ---"
|
||||
bd compact --dry-run --all
|
||||
|
||||
echo
|
||||
read -p "Proceed with Tier 1 compaction? (y/N) " -n 1 -r
|
||||
echo
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
echo "--- Running Tier 1 Compaction ---"
|
||||
bd compact --all
|
||||
echo "✅ Tier 1 compaction complete"
|
||||
else
|
||||
echo "⏭️ Skipping Tier 1"
|
||||
fi
|
||||
|
||||
# Preview Tier 2
|
||||
echo
|
||||
echo "--- Preview Tier 2 Candidates ---"
|
||||
bd compact --dry-run --all --tier 2
|
||||
|
||||
echo
|
||||
read -p "Proceed with Tier 2 compaction? (y/N) " -n 1 -r
|
||||
echo
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
echo "--- Running Tier 2 Compaction ---"
|
||||
bd compact --all --tier 2
|
||||
echo "✅ Tier 2 compaction complete"
|
||||
else
|
||||
echo "⏭️ Skipping Tier 2"
|
||||
fi
|
||||
|
||||
# Show stats
|
||||
echo
|
||||
echo "--- Final Statistics ---"
|
||||
bd compact --stats
|
||||
|
||||
echo
|
||||
echo "=== Compaction Complete ==="
|
||||
echo
|
||||
echo "Next steps:"
|
||||
echo " 1. Review compacted issues: bd list --json | jq '.[] | select(.compaction_level > 0)'"
|
||||
echo " 2. Commit changes: git add .beads/issues.jsonl issues.db && git commit -m 'Compact old issues'"
|
||||
echo " 3. Push to remote: git push"
|
||||
Reference in New Issue
Block a user