johno/nixos-configs
1
0
Fork 0
Code Issues 3 Pull Requests 8 Actions Packages Projects Releases Wiki Activity

Update plan to use WinBtrfs

Browse Source
This commit is contained in:
John Ogle
2025-07-29 11:18:24 -07:00
parent ffd7ce45a7
commit 433cfe3985
1 changed files with 230 additions and 145 deletions
Download Patch File Download Diff File Expand all files Collapse all files

375
nixos-steam-dual-boot-plan.md
View File

@@ -1,4 +1,4 @@
# NixOS Steam Dual Boot Implementation Plan
# NixOS Steam Dual Boot Implementation Plan (Btrfs + Beesd Edition)
## Goals & Motivations
@@ -12,10 +12,11 @@
- **Seamless game access**: Games should be available on both OSes without manual intervention
- **Update compatibility**: Game updates from either OS should be usable by both
- **Future-proof architecture**: Solution should be extensible and maintainable
- **Multi-user support**: Handle 3 Windows users sharing libraries, with 1 user also using Linux
## Architectural Overview
### The Problem with Traditional Approaches
### The Problem with Previous Approaches
**Shared NTFS Library (Traditional)**:
- ❌ Proton creates files with colons, corrupting NTFS
@@ -23,226 +24,310 @@
- ❌ Permission issues plague the setup
- ❌ Valve officially discourages this approach
**Complex Symlink Management**:
- ❌ Requires custom scripts and maintenance
- ❌ Fragile edge cases with Steam updates
- ❌ Potential compatibility issues with Proton and anti-cheat
**Separate Libraries**:
- ❌ Wastes terabytes of storage
- ❌ Games must be installed twice
- ❌ No benefit from either OS's installations
### Our Solution: Asymmetric Symlink Architecture
### Our Solution: Btrfs + Beesd Automatic Deduplication
```
Windows Steam → NTFS Shared Library (real files, primary storage)
↑
Linux Steam → ext4 Library (symlinks) ──┘
Windows Steam → /steam/windows/ (btrfs via WinBtrfs driver)
↓
Linux Steam → /steam/linux/ (native btrfs)
↓
[beesd automatically deduplicates identical files]
```
**Key Insight**: Make Linux the "smart" side that handles complexity, while Windows gets simple, reliable access to real files.
**Key Insight**: Let each Steam installation work independently, rely on proven btrfs deduplication technology to eliminate duplicate storage automatically.
## Architecture Deep Dive
### Component 1: NTFS Shared Library
### Component 1: Shared Btrfs Filesystem
**Purpose**: Primary storage for all game files, designed for Windows Steam
- **Location**: `/mnt/shared-steam` (mounted NTFS partition)
- **Ownership**: Windows Steam has native, unrestricted access
- **Contents**: Real game files, installed/updated by Windows Steam
**Purpose**: Single high-performance filesystem for all game storage
- **Location**: `/steam` (dedicated btrfs partition)
- **Contents**:
- `/steam/windows/` - Windows Steam library
- `/steam/linux/` - Linux Steam library
- **Access**: Native on Linux, WinBtrfs driver on Windows
**Why this satisfies our goals**:
- ✅ **Windows stability**: Native NTFS access, no drivers or special handling
- ✅ **Performance**: No filesystem translation layer for Windows
- ✅ **Reliability**: Windows Steam operates in its natural environment
- ✅ **Automatic deduplication**: Beesd handles duplicate elimination transparently
- ✅ **No maintenance**: Zero custom scripts or symlink management
- ✅ **Independent operation**: Each Steam installation works normally
- ✅ **Performance**: Optimized mount options for gaming workloads
### Component 2: Linux Native Library
### Component 2: Beesd Deduplication Service
**Purpose**: Linux Steam's working directory with intelligent file management
- **Location**: `/home/user/.local/share/Steam/steamapps` (ext4/btrfs)
- **Contents**: Symlinks to shared library + Linux-specific metadata
- **Management**: Automated script maintains symlink consistency
**Purpose**: Automatic background deduplication of game files
**Why this satisfies our goals**:
- ✅ **Eliminate storage waste**: Symlinks use negligible space
- ✅ **Linux performance**: Native filesystem for Steam client data
- ✅ **Graceful degradation**: Linux games still work if shared library fails
**Core Functionality**:
- Continuously scans `/steam` for duplicate blocks
- Automatically deduplicates identical files between `/steam/windows/` and `/steam/linux/`
- Operates transparently - games never know deduplication is happening
- Handles common duplicates: DirectX runtimes, Visual C++ redistributables, game engines, shared assets
### Component 3: Intelligent Deduplication Script
**Expected Efficiency**:
- **40-70% storage savings** for typical game libraries
- **Common targets**: Unity/Unreal engine files, shared libraries, identical texture assets
- **Real-time operation**: New duplicates eliminated automatically
**Purpose**: Automatically manage game installations and eliminate duplicates
### Component 3: WinBtrfs Driver Integration
**Core Logic**:
```bash
# Scenario 1: New Windows install
if [[ -d "$shared_path" && ! -e "$linux_path" ]]; then
ln -s "$shared_path" "$linux_path" # Link to Windows install
fi
# Scenario 2: Linux install exists, no Windows version
if [[ -d "$linux_path" && ! -L "$linux_path" && ! -d "$shared_path" ]]; then
mv "$linux_path" "$shared_path" # Move to shared storage
ln -s "$shared_path" "$linux_path" # Replace with symlink
fi
# Scenario 3: Both exist (duplicate install)
if [[ -d "$linux_path" && ! -L "$linux_path" && -d "$shared_path" ]]; then
# Intelligent merge: keep larger/newer, discard duplicate
deduplicate_installation "$game_name"
fi
```
**Why this satisfies our goals**:
- ✅ **Minimize maintenance**: Fully automated, runs on boot/schedule
- ✅ **Eliminate waste**: Automatically consolidates duplicate installs
- ✅ **Seamless access**: New games appear on both OSes transparently
### Component 4: NTFS Mount Configuration
**Purpose**: Secure, performant NTFS access with proper permissions
**Purpose**: Provide Windows with native btrfs read/write access
```nix
fileSystems."/mnt/shared-steam" = {
device = "/dev/disk/by-uuid/YOUR-NTFS-UUID";
fsType = "ntfs-3g";
# No special configuration needed - standard btrfs mount
fileSystems."/steam" = {
device = "/dev/disk/by-uuid/YOUR-BTRFS-UUID";
fsType = "btrfs";
options = [
"uid=1000" # Your user owns all files
"gid=100" # Users group
"umask=022" # Secure but usable permissions
"dmask=022" # Directory permissions: 755
"fmask=133" # File permissions: 644 + executable
"windows_names" # Handle Windows naming conventions
"big_writes" # Performance optimization
"user_xattr" # Extended attributes for compatibility
"noatime" # Don't update access times - major gaming performance boost
"ssd" # SSD optimizations
];
};
```
**Why this satisfies our goals**:
- ✅ **Eliminate permission issues**: Consistent ownership and permissions
- ✅ **Performance**: Optimized mount options for gaming workloads
- ✅ **Reliability**: Stable, well-tested NTFS-3G configuration
**Windows Requirements**:
- Install WinBtrfs driver (https://github.com/maharmstone/btrfs)
- Configure Steam library path to point to `/steam/windows/`
- No special configuration needed - works like any other drive
### Component 4: Beesd Configuration
**Purpose**: Optimized deduplication for gaming workloads
```nix
services.beesd.filesystems = {
steam = {
spec = "/steam";
hashTableSizeMB = 512; # Sized for ~4TB of game data (128MB per TB)
verbosity = "err"; # Only show actual problems
};
};
```
**Hash Table Storage**:
- Stored as file: `/steam/.beeshash`
- 512MB handles up to 4TB of game data efficiently (beesd recommends 128MB per TB)
- Loaded into RAM on-demand, not all resident simultaneously
- Smaller hash table = better performance and less disk usage
## Partition Strategy
### 4-Partition Architecture
1. **Windows Boot Partition** (NTFS) - *Existing*
- Standard Windows system drive
- Unchanged from current setup
2. **Linux Boot Partition** (btrfs/ext4) - *New*
- NixOS system installation
- Standard Linux root filesystem
3. **Shared Steam Library** (btrfs) - *New*
- `/steam/windows/` - Windows Steam library
- `/steam/linux/` - Linux Steam library
- Automatic beesd deduplication
4. **Legacy Windows Steam** (NTFS) - *Existing, Optional*
- Keep for anti-cheat games that may not work on btrfs
- Can be eliminated if all games work on btrfs
- Provides fallback option during testing
### Multi-User Considerations
**Current Setup**: 3 Windows users sharing Steam libraries
**Migration Strategy**:
- Other Windows users continue using existing NTFS library
- Primary user (johno) experiments with btrfs library
- Easy rollback: point Steam back to NTFS if issues arise
- Gradual migration as confidence builds
## Implementation Strategy
### Phase 1: Base Setup
1. **Partition Management**:
- Identify/create NTFS partition for shared library
- Ensure sufficient space (recommend 1TB+ for modern libraries)
1. **Partition Creation**:
- Create new btrfs partition for `/steam` (recommend 2TB+ for modern libraries)
- Install NixOS on separate Linux boot partition
- Keep existing Windows partitions untouched
2. **NixOS Configuration**:
- Add NTFS mount with optimized options
- Enable Steam with proper permissions
- Configure automatic script execution
```nix
fileSystems."/steam" = {
device = "/dev/disk/by-uuid/YOUR-BTRFS-UUID";
fsType = "btrfs";
options = [ "noatime" "ssd" ];
};
3. **Initial Testing**:
- Install test game on Windows
- Verify Linux can access via symlink
- Confirm updates work from both sides
services.beesd.filesystems = {
steam = {
spec = "/steam";
hashTableSizeMB = 512; # 128MB per TB recommended
verbosity = "err";
};
};
```
### Phase 2: Script Development
3. **Windows Setup**:
- Install WinBtrfs driver
- Add Steam library pointing to `/steam/windows/`
- Test with a few small games initially
1. **Core Deduplication Logic**:
- Game discovery and inventory
- Symlink management
- Conflict resolution
### Phase 2: Gradual Migration
2. **Safety Features**:
- Permission verification
- Backup creation before major operations
- Rollback capability
1. **Safe Game Testing**:
- Start with single-player games
- Test save game compatibility
- Verify performance matches NTFS installation
3. **Automation Integration**:
- SystemD service for boot-time execution
- Optional inotify-based real-time updates
2. **Anti-Cheat Evaluation**:
- Test multiplayer games progressively
- Document which games work on btrfs vs require NTFS
- Keep problematic games on legacy NTFS partition
3. **Deduplication Verification**:
- Monitor beesd logs for successful deduplication
- Use `btrfs filesystem du /steam` to verify space savings
- Benchmark game loading times vs separate installations
### Phase 3: Optimization
1. **Performance Tuning**:
- I/O scheduler optimization for gaming workloads
- NTFS mount parameter fine-tuning
- Script performance optimization
- Monitor btrfs performance under gaming workloads
- Adjust mount options if needed
- Optimize beesd parameters based on actual usage
2. **Robustness Improvements**:
- Error handling and recovery
- Logging and monitoring
- Edge case handling
2. **Monitoring Setup**:
- SystemD service monitoring for beesd
- Disk space alerts for `/steam` partition
- Basic health checks for WinBtrfs stability
## Trade-offs and Considerations
### Advantages
**Storage Efficiency**:
- ✅ Single copy of each game (terabytes saved)
- ✅ Symlinks use negligible space
- ✅ No artificial storage constraints
**Maintenance**:
- ✅ Fully automated after initial setup
- ✅ Self-healing: script fixes broken symlinks
- ✅ 40-70% storage savings through automatic deduplication
- ✅ No manual intervention required
- ✅ Works with any game installation method
**Compatibility**:
- ✅ Windows Steam operates normally
- ✅ Linux Steam gets full game access
- ✅ Updates from either OS benefit both
**Simplicity**:
- ✅ No custom scripts to maintain
- ✅ No symlink complexity
- ✅ Standard Steam library management on both OSes
**Reliability**:
- ✅ Each Steam installation completely independent
- ✅ Btrfs and beesd are mature, proven technologies
- ✅ Graceful degradation if deduplication fails
**Performance**:
- ✅ Optimized mount options for gaming
- ✅ No compression overhead
- ✅ Native filesystem performance on both OSes
### Limitations
**Complexity**:
- ⚠️ More complex than separate libraries
- ⚠️ Requires custom script maintenance
- ⚠️ NixOS-specific configuration
**Windows Dependencies**:
- ⚠️ Requires WinBtrfs third-party driver
- ⚠️ Driver updates needed with major Windows releases
- ⚠️ Potential compatibility issues with some anti-cheat systems
**Dependencies**:
- ⚠️ Relies on NTFS-3G stability
- ⚠️ Script must run reliably
- ⚠️ NTFS partition must remain healthy
**Complexity Trade-offs**:
- ⚠️ More partitions to manage than single-filesystem approach
- ⚠️ Beesd adds background CPU/disk usage (minimal but present)
- ⚠️ Hash table requires disk space (512MB for large libraries)
**Edge Cases**:
- ⚠️ Some games may have platform-specific files
- ⚠️ Very large libraries may stress script performance
- ⚠️ Symlink chains could confuse some games
**Platform Compatibility**:
- ⚠️ Some games may prefer NTFS for maximum compatibility
- ⚠️ Anti-cheat systems may flag non-NTFS installations
- ⚠️ WinBtrfs stability depends on third-party development
### Risk Mitigation
**Backup Strategy**:
- Regular snapshots of shared library
- Script creates backups before major operations
- Steam's built-in backup/restore as fallback
- Regular btrfs snapshots of `/steam` partition
- Keep legacy NTFS Steam library as fallback
- Steam's built-in backup/restore for critical games
**Fallback Options**:
- Linux can install games locally if shared library fails
- Windows operates independently
- Script can be disabled without breaking either Steam installation
- Easy to revert games to NTFS library if needed
- Linux can install games locally if `/steam` partition fails
- Independent operation means failure in one OS doesn't affect the other
**Monitoring**:
- Log all script operations
- SystemD service status monitoring
- Disk space monitoring for both partitions
- SystemD service status for beesd
- Disk space monitoring for early warning
- Game launch testing after major updates
## Expected Outcomes
### Immediate Benefits
- **50-80% storage savings** (typical for large game libraries)
- **40-70% storage savings** for typical game libraries
- **Zero maintenance** after initial setup
- **Identical game performance** on both platforms
- **Identical performance** to native installations
- **Future-proof** architecture using standard technologies
### Long-term Benefits
- **Simplified game management**: Install once, play anywhere
- **Faster OS switching**: No need to reinstall games when switching primary OS
- **Future extensibility**: Foundation for more advanced features (save sync, etc.)
- **Automatic optimization**: New games deduplicated without intervention
- **Simplified management**: One shared library instead of separate installations
- **Technology leverage**: Benefits from ongoing btrfs and beesd improvements
### Success Metrics
- ✅ Games launch successfully from both OSes
- ✅ Updates from either OS work correctly
- ✅ No manual intervention required for normal operation
- ✅ Storage usage comparable to single-OS setup
- ✅ No performance degradation compared to native installs
- ✅ Save games work correctly on both platforms
- ✅ Updates from either OS don't break the other
- ✅ Storage usage 40-70% less than separate libraries
- ✅ No performance degradation vs native installations
- ✅ Anti-cheat compatibility acceptable for target games
### Monitoring and Maintenance
**Automated Monitoring**:
```nix
# Add to NixOS configuration for basic monitoring
systemd.services.steam-health-check = {
description = "Check Steam partition and beesd health";
serviceConfig = {
Type = "oneshot";
ExecStart = pkgs.writeScript "steam-health" ''
#!/bin/bash
# Check beesd service status
systemctl is-active beesd-steam >/dev/null || echo "WARN: beesd not running"
# Check disk space
USAGE=$(df /steam | tail -1 | awk '{print $5}' | sed 's/%//')
[ "$USAGE" -gt 90 ] && echo "WARN: Steam partition >90% full"
# Verify both directories exist
[ ! -d "/steam/windows" ] && echo "ERROR: Windows Steam directory missing"
[ ! -d "/steam/linux" ] && echo "ERROR: Linux Steam directory missing"
'';
};
};
systemd.timers.steam-health-check = {
wantedBy = [ "timers.target" ];
timerConfig = {
OnCalendar = "daily";
Persistent = true;
};
};
```
## Implementation Files Needed
1. **NixOS Configuration Module** (`steam-dual-boot.nix`)
2. **Deduplication Script** (`steam-dedupe.sh`)
3. **SystemD Service Definition** (`steam-linker.service`)
4. **Installation Instructions** (`SETUP.md`)
5. **Troubleshooting Guide** (`TROUBLESHOOTING.md`)
1. **NixOS Configuration Addition** - Add to existing machine config
2. **WinBtrfs Installation Guide** - Windows setup instructions
3. **Migration Checklist** - Step-by-step game migration process
4. **Troubleshooting Guide** - Common issues and solutions
This architecture provides a robust, maintainable solution that maximizes storage efficiency while preserving the stability and performance requirements of both operating systems.
This architecture provides maximum storage efficiency with minimal complexity, leveraging proven technologies instead of custom solutions. The automatic nature of btrfs deduplication eliminates the maintenance overhead of complex symlink management while providing excellent storage savings.