Compare commits

..

2 Commits

Author SHA1 Message Date
433cfe3985 Update plan to use WinBtrfs 2025-07-29 11:18:24 -07:00
ffd7ce45a7 Initial plan using symlinks 2025-07-29 10:34:50 -07:00
4 changed files with 335 additions and 238 deletions

View File

@@ -39,15 +39,6 @@
boot.kernelPackages = pkgs.linuxPackages_latest;
# Btrfs deduplication service
services.beesd.filesystems = {
root = {
spec = "/";
hashTableSizeMB = 32; # 128MB per TB recommended, ~225GB = ~32MB
verbosity = "err"; # Only show actual problems
};
};
# Enable networking
networking.networkmanager.enable = true;

View File

@@ -14,12 +14,8 @@
boot.extraModulePackages = [ ];
fileSystems."/" =
{ device = "/dev/disk/by-uuid/223a44e5-91e2-4272-830e-129166042a1d";
fsType = "btrfs";
options = [
"compress=zstd" # Enable zstd compression for space savings
"noatime" # Don't update access times for performance
];
{ device = "/dev/disk/by-uuid/bd396529-e2c4-47cb-b844-8d6ed841f81a";
fsType = "ext4";
};
boot.initrd.luks.devices."luks-4126fbd4-bd09-4ece-af0d-6fff414c21b3".device = "/dev/disk/by-uuid/4126fbd4-bd09-4ece-af0d-6fff414c21b3";

View File

@@ -1,223 +0,0 @@
# NixBook ext4 to btrfs Migration Guide
## Overview
This guide converts your nixbook machine from ext4 to btrfs with zstd compression and beesd deduplication while preserving your LUKS encryption and all data.
## Current System Info
- **Hostname**: nix-book
- **Root filesystem**: ext4 on `/dev/disk/by-uuid/bd396529-e2c4-47cb-b844-8d6ed841f81a`
- **Encryption**: LUKS with two devices configured
- **Current usage**: 138GB used / 225GB total (65% full)
- **Free space**: 76GB available (sufficient for conversion)
## Pre-Migration Checklist
### 1. Create Full System Backup (CRITICAL)
```bash
# Boot from NixOS live USB
# Mount encrypted filesystem
cryptsetup luksOpen /dev/disk/by-uuid/4126fbd4-bd09-4ece-af0d-6fff414c21b3 luks-nixbook
mount /dev/mapper/luks-nixbook /mnt
# Create backup to external drive (adjust target as needed)
rsync -avxHAX --progress /mnt/ /path/to/backup/nixbook-backup/
```
### 2. Verify Configuration Changes
The following files have been updated for btrfs:
- `machines/nix-book/configuration.nix` - Added beesd service
- `machines/nix-book/hardware-configuration.nix` - Changed fsType to btrfs with compression
## Migration Process
### Phase 1: Boot to Live Environment
1. **Create NixOS live USB**:
```bash
# Download latest NixOS ISO
# Flash to USB drive
dd if=nixos-minimal-xx.xx-x86_64-linux.iso of=/dev/sdX bs=4M status=progress
```
2. **Boot from live USB** and ensure you can access the encrypted drives
### Phase 2: Filesystem Conversion
3. **Unlock LUKS volumes**:
```bash
cryptsetup luksOpen /dev/disk/by-uuid/4126fbd4-bd09-4ece-af0d-6fff414c21b3 luks-nixbook
cryptsetup luksOpen /dev/disk/by-uuid/b614167b-9045-4234-a441-ac6f60a96d81 luks-nixbook2
```
4. **Check filesystem before conversion**:
```bash
fsck.ext4 -f /dev/mapper/luks-nixbook
```
5. **Convert ext4 to btrfs** (this preserves all data):
```bash
# Install btrfs-progs if not available
nix-shell -p btrfs-progs
# Convert the filesystem (takes 15-45 minutes depending on data)
btrfs-convert /dev/mapper/luks-nixbook
# Verify conversion succeeded
mount /dev/mapper/luks-nixbook /mnt
ls -la /mnt # Should show your normal filesystem
btrfs filesystem show /mnt
```
6. **Get new filesystem UUID** (may have changed):
```bash
blkid /dev/mapper/luks-nixbook
# Note the new UUID if it changed
```
### Phase 3: Configuration Update
7. **Mount and chroot into system**:
```bash
mount -o compress=zstd,noatime /dev/mapper/luks-nixbook /mnt
mount /dev/disk/by-uuid/7A0B-CF88 /mnt/boot
nixos-enter --root /mnt
```
8. **Update hardware-configuration.nix** if UUID changed:
```bash
# Edit /etc/nixos/hardware-configuration.nix if needed
# Update the UUID in fileSystems."/" section
```
9. **Rebuild system with btrfs configuration**:
```bash
cd /home/johno/nixos-configs
nixos-rebuild switch --flake .#nix-book
```
### Phase 4: Enable Compression and Deduplication
10. **Reboot into new btrfs system**:
```bash
exit # Exit chroot
umount -R /mnt
reboot
```
11. **Verify btrfs is working**:
```bash
mount | grep btrfs
btrfs filesystem usage /
```
12. **Enable and start beesd**:
```bash
systemctl status beesd-root
systemctl start beesd-root
systemctl enable beesd-root
```
13. **Force compression on existing files** (optional but recommended):
```bash
# This will compress existing files with zstd
btrfs filesystem defragment -r -czstd /
```
## Post-Migration Verification
### Check System Health
```bash
# Verify btrfs health
btrfs scrub start /
btrfs scrub status /
# Check compression effectiveness
compsize /
# Monitor beesd deduplication
journalctl -u beesd-root -f
# Check filesystem usage
btrfs filesystem usage /
df -h /
```
### Performance Monitoring
```bash
# Monitor beesd hash table
ls -lh /.beeshash
# Check compression ratio over time
compsize /home /nix /var
```
## Expected Benefits
### Space Savings
- **Compression**: 20-30% reduction in disk usage from zstd
- **Deduplication**: Additional 10-20% savings on duplicate files
- **Combined**: Potentially 30-40% total space savings
### Performance Impact
- **Compression**: Minimal CPU overhead, often improves I/O performance
- **Deduplication**: Background process, minimal impact during normal use
- **Overall**: Should be neutral to positive performance impact
## Rollback Plan (Emergency)
If something goes wrong:
1. **Boot from live USB**
2. **Restore from backup**:
```bash
cryptsetup luksOpen /dev/disk/by-uuid/4126fbd4-bd09-4ece-af0d-6fff414c21b3 luks-nixbook
mkfs.ext4 /dev/mapper/luks-nixbook
mount /dev/mapper/luks-nixbook /mnt
rsync -avxHAX --progress /path/to/backup/nixbook-backup/ /mnt/
```
3. **Restore original hardware-configuration.nix** with ext4 settings
4. **Rebuild and reboot**
## Troubleshooting
### Common Issues
**"Device busy" during conversion**:
- Ensure no processes are accessing the filesystem
- Check with `lsof` and `fuser`
**UUID changed after conversion**:
- Update hardware-configuration.nix with new UUID
- Regenerate initrd: `nixos-rebuild switch`
**Beesd service fails to start**:
- Check disk space for hash table
- Verify filesystem is btrfs: `mount | grep btrfs`
- Check logs: `journalctl -u beesd-root`
**Boot issues after conversion**:
- Boot from live USB
- Check /boot partition is mounted correctly
- Verify LUKS UUIDs match in configuration
- Rebuild bootloader: `nixos-rebuild switch --install-bootloader`
## Maintenance
### Regular Tasks
```bash
# Monthly scrub (checks for corruption)
btrfs scrub start /
# Monitor compression effectiveness
compsize /
# Check beesd deduplication status
systemctl status beesd-root
```
### Space Management
```bash
# Balance filesystem (defragments and optimizes)
btrfs balance start -dusage=50 /
# Check for space issues
btrfs filesystem usage /
```
This migration preserves all your data while gaining the benefits of modern btrfs features including transparent compression and automatic deduplication.

View File

@@ -0,0 +1,333 @@
# NixOS Steam Dual Boot Implementation Plan (Btrfs + Beesd Edition)
## Goals & Motivations
### Primary Goals
1. **Eliminate storage waste**: Avoid duplicating terabytes of Steam games across Windows and NixOS
2. **Minimize maintenance overhead**: Create a solution that works reliably without constant tweaking
3. **Preserve Windows stability**: Ensure Windows Steam functionality remains unaffected by the dual-boot setup
4. **Maintain gaming performance**: No significant performance degradation on either OS
### Secondary Goals
- **Seamless game access**: Games should be available on both OSes without manual intervention
- **Update compatibility**: Game updates from either OS should be usable by both
- **Future-proof architecture**: Solution should be extensible and maintainable
- **Multi-user support**: Handle 3 Windows users sharing libraries, with 1 user also using Linux
## Architectural Overview
### The Problem with Previous Approaches
**Shared NTFS Library (Traditional)**:
- ❌ Proton creates files with colons, corrupting NTFS
- ❌ Requires fragile symlinks that Windows can break
- ❌ Permission issues plague the setup
- ❌ Valve officially discourages this approach
**Complex Symlink Management**:
- ❌ Requires custom scripts and maintenance
- ❌ Fragile edge cases with Steam updates
- ❌ Potential compatibility issues with Proton and anti-cheat
**Separate Libraries**:
- ❌ Wastes terabytes of storage
- ❌ Games must be installed twice
- ❌ No benefit from either OS's installations
### Our Solution: Btrfs + Beesd Automatic Deduplication
```
Windows Steam → /steam/windows/ (btrfs via WinBtrfs driver)
Linux Steam → /steam/linux/ (native btrfs)
[beesd automatically deduplicates identical files]
```
**Key Insight**: Let each Steam installation work independently, rely on proven btrfs deduplication technology to eliminate duplicate storage automatically.
## Architecture Deep Dive
### Component 1: Shared Btrfs Filesystem
**Purpose**: Single high-performance filesystem for all game storage
- **Location**: `/steam` (dedicated btrfs partition)
- **Contents**:
- `/steam/windows/` - Windows Steam library
- `/steam/linux/` - Linux Steam library
- **Access**: Native on Linux, WinBtrfs driver on Windows
**Why this satisfies our goals**:
-**Automatic deduplication**: Beesd handles duplicate elimination transparently
-**No maintenance**: Zero custom scripts or symlink management
-**Independent operation**: Each Steam installation works normally
-**Performance**: Optimized mount options for gaming workloads
### Component 2: Beesd Deduplication Service
**Purpose**: Automatic background deduplication of game files
**Core Functionality**:
- Continuously scans `/steam` for duplicate blocks
- Automatically deduplicates identical files between `/steam/windows/` and `/steam/linux/`
- Operates transparently - games never know deduplication is happening
- Handles common duplicates: DirectX runtimes, Visual C++ redistributables, game engines, shared assets
**Expected Efficiency**:
- **40-70% storage savings** for typical game libraries
- **Common targets**: Unity/Unreal engine files, shared libraries, identical texture assets
- **Real-time operation**: New duplicates eliminated automatically
### Component 3: WinBtrfs Driver Integration
**Purpose**: Provide Windows with native btrfs read/write access
```nix
# No special configuration needed - standard btrfs mount
fileSystems."/steam" = {
device = "/dev/disk/by-uuid/YOUR-BTRFS-UUID";
fsType = "btrfs";
options = [
"noatime" # Don't update access times - major gaming performance boost
"ssd" # SSD optimizations
];
};
```
**Windows Requirements**:
- Install WinBtrfs driver (https://github.com/maharmstone/btrfs)
- Configure Steam library path to point to `/steam/windows/`
- No special configuration needed - works like any other drive
### Component 4: Beesd Configuration
**Purpose**: Optimized deduplication for gaming workloads
```nix
services.beesd.filesystems = {
steam = {
spec = "/steam";
hashTableSizeMB = 512; # Sized for ~4TB of game data (128MB per TB)
verbosity = "err"; # Only show actual problems
};
};
```
**Hash Table Storage**:
- Stored as file: `/steam/.beeshash`
- 512MB handles up to 4TB of game data efficiently (beesd recommends 128MB per TB)
- Loaded into RAM on-demand, not all resident simultaneously
- Smaller hash table = better performance and less disk usage
## Partition Strategy
### 4-Partition Architecture
1. **Windows Boot Partition** (NTFS) - *Existing*
- Standard Windows system drive
- Unchanged from current setup
2. **Linux Boot Partition** (btrfs/ext4) - *New*
- NixOS system installation
- Standard Linux root filesystem
3. **Shared Steam Library** (btrfs) - *New*
- `/steam/windows/` - Windows Steam library
- `/steam/linux/` - Linux Steam library
- Automatic beesd deduplication
4. **Legacy Windows Steam** (NTFS) - *Existing, Optional*
- Keep for anti-cheat games that may not work on btrfs
- Can be eliminated if all games work on btrfs
- Provides fallback option during testing
### Multi-User Considerations
**Current Setup**: 3 Windows users sharing Steam libraries
**Migration Strategy**:
- Other Windows users continue using existing NTFS library
- Primary user (johno) experiments with btrfs library
- Easy rollback: point Steam back to NTFS if issues arise
- Gradual migration as confidence builds
## Implementation Strategy
### Phase 1: Base Setup
1. **Partition Creation**:
- Create new btrfs partition for `/steam` (recommend 2TB+ for modern libraries)
- Install NixOS on separate Linux boot partition
- Keep existing Windows partitions untouched
2. **NixOS Configuration**:
```nix
fileSystems."/steam" = {
device = "/dev/disk/by-uuid/YOUR-BTRFS-UUID";
fsType = "btrfs";
options = [ "noatime" "ssd" ];
};
services.beesd.filesystems = {
steam = {
spec = "/steam";
hashTableSizeMB = 512; # 128MB per TB recommended
verbosity = "err";
};
};
```
3. **Windows Setup**:
- Install WinBtrfs driver
- Add Steam library pointing to `/steam/windows/`
- Test with a few small games initially
### Phase 2: Gradual Migration
1. **Safe Game Testing**:
- Start with single-player games
- Test save game compatibility
- Verify performance matches NTFS installation
2. **Anti-Cheat Evaluation**:
- Test multiplayer games progressively
- Document which games work on btrfs vs require NTFS
- Keep problematic games on legacy NTFS partition
3. **Deduplication Verification**:
- Monitor beesd logs for successful deduplication
- Use `btrfs filesystem du /steam` to verify space savings
- Benchmark game loading times vs separate installations
### Phase 3: Optimization
1. **Performance Tuning**:
- Monitor btrfs performance under gaming workloads
- Adjust mount options if needed
- Optimize beesd parameters based on actual usage
2. **Monitoring Setup**:
- SystemD service monitoring for beesd
- Disk space alerts for `/steam` partition
- Basic health checks for WinBtrfs stability
## Trade-offs and Considerations
### Advantages
**Storage Efficiency**:
- ✅ 40-70% storage savings through automatic deduplication
- ✅ No manual intervention required
- ✅ Works with any game installation method
**Simplicity**:
- ✅ No custom scripts to maintain
- ✅ No symlink complexity
- ✅ Standard Steam library management on both OSes
**Reliability**:
- ✅ Each Steam installation completely independent
- ✅ Btrfs and beesd are mature, proven technologies
- ✅ Graceful degradation if deduplication fails
**Performance**:
- ✅ Optimized mount options for gaming
- ✅ No compression overhead
- ✅ Native filesystem performance on both OSes
### Limitations
**Windows Dependencies**:
- ⚠️ Requires WinBtrfs third-party driver
- ⚠️ Driver updates needed with major Windows releases
- ⚠️ Potential compatibility issues with some anti-cheat systems
**Complexity Trade-offs**:
- ⚠️ More partitions to manage than single-filesystem approach
- ⚠️ Beesd adds background CPU/disk usage (minimal but present)
- ⚠️ Hash table requires disk space (512MB for large libraries)
**Platform Compatibility**:
- ⚠️ Some games may prefer NTFS for maximum compatibility
- ⚠️ Anti-cheat systems may flag non-NTFS installations
- ⚠️ WinBtrfs stability depends on third-party development
### Risk Mitigation
**Backup Strategy**:
- Regular btrfs snapshots of `/steam` partition
- Keep legacy NTFS Steam library as fallback
- Steam's built-in backup/restore for critical games
**Fallback Options**:
- Easy to revert games to NTFS library if needed
- Linux can install games locally if `/steam` partition fails
- Independent operation means failure in one OS doesn't affect the other
**Monitoring**:
- SystemD service status for beesd
- Disk space monitoring for early warning
- Game launch testing after major updates
## Expected Outcomes
### Immediate Benefits
- **40-70% storage savings** for typical game libraries
- **Zero maintenance** after initial setup
- **Identical performance** to native installations
- **Future-proof** architecture using standard technologies
### Long-term Benefits
- **Automatic optimization**: New games deduplicated without intervention
- **Simplified management**: One shared library instead of separate installations
- **Technology leverage**: Benefits from ongoing btrfs and beesd improvements
### Success Metrics
- ✅ Games launch successfully from both OSes
- ✅ Save games work correctly on both platforms
- ✅ Updates from either OS don't break the other
- ✅ Storage usage 40-70% less than separate libraries
- ✅ No performance degradation vs native installations
- ✅ Anti-cheat compatibility acceptable for target games
### Monitoring and Maintenance
**Automated Monitoring**:
```nix
# Add to NixOS configuration for basic monitoring
systemd.services.steam-health-check = {
description = "Check Steam partition and beesd health";
serviceConfig = {
Type = "oneshot";
ExecStart = pkgs.writeScript "steam-health" ''
#!/bin/bash
# Check beesd service status
systemctl is-active beesd-steam >/dev/null || echo "WARN: beesd not running"
# Check disk space
USAGE=$(df /steam | tail -1 | awk '{print $5}' | sed 's/%//')
[ "$USAGE" -gt 90 ] && echo "WARN: Steam partition >90% full"
# Verify both directories exist
[ ! -d "/steam/windows" ] && echo "ERROR: Windows Steam directory missing"
[ ! -d "/steam/linux" ] && echo "ERROR: Linux Steam directory missing"
'';
};
};
systemd.timers.steam-health-check = {
wantedBy = [ "timers.target" ];
timerConfig = {
OnCalendar = "daily";
Persistent = true;
};
};
```
## Implementation Files Needed
1. **NixOS Configuration Addition** - Add to existing machine config
2. **WinBtrfs Installation Guide** - Windows setup instructions
3. **Migration Checklist** - Step-by-step game migration process
4. **Troubleshooting Guide** - Common issues and solutions
This architecture provides maximum storage efficiency with minimal complexity, leveraging proven technologies instead of custom solutions. The automatic nature of btrfs deduplication eliminates the maintenance overhead of complex symlink management while providing excellent storage savings.