johno/nixos-configs
1
0
Fork 0
Code Issues 3 Pull Requests 8 Actions Packages Projects Releases Wiki Activity

Update plan to use WinBtrfs

Browse Source
This commit is contained in:
John Ogle
2025-07-29 11:18:24 -07:00
parent ffd7ce45a7
commit 433cfe3985
1 changed files with 230 additions and 145 deletions
Download Patch File Download Diff File Expand all files Collapse all files

375
nixos-steam-dual-boot-plan.md
View File

@@ -1,4 +1,4 @@
# NixOS Steam Dual Boot Implementation Plan # NixOS Steam Dual Boot Implementation Plan (Btrfs + Beesd Edition)
## Goals & Motivations ## Goals & Motivations
@@ -12,10 +12,11 @@
- **Seamless game access**: Games should be available on both OSes without manual intervention - **Seamless game access**: Games should be available on both OSes without manual intervention
- **Update compatibility**: Game updates from either OS should be usable by both - **Update compatibility**: Game updates from either OS should be usable by both
- **Future-proof architecture**: Solution should be extensible and maintainable - **Future-proof architecture**: Solution should be extensible and maintainable
- **Multi-user support**: Handle 3 Windows users sharing libraries, with 1 user also using Linux
## Architectural Overview ## Architectural Overview
### The Problem with Traditional Approaches ### The Problem with Previous Approaches
**Shared NTFS Library (Traditional)**: **Shared NTFS Library (Traditional)**:
- ❌ Proton creates files with colons, corrupting NTFS - ❌ Proton creates files with colons, corrupting NTFS
@@ -23,226 +24,310 @@
- ❌ Permission issues plague the setup - ❌ Permission issues plague the setup
- ❌ Valve officially discourages this approach - ❌ Valve officially discourages this approach
**Complex Symlink Management**:
- ❌ Requires custom scripts and maintenance
- ❌ Fragile edge cases with Steam updates
- ❌ Potential compatibility issues with Proton and anti-cheat
**Separate Libraries**: **Separate Libraries**:
- ❌ Wastes terabytes of storage - ❌ Wastes terabytes of storage
- ❌ Games must be installed twice - ❌ Games must be installed twice
- ❌ No benefit from either OS's installations - ❌ No benefit from either OS's installations
### Our Solution: Asymmetric Symlink Architecture ### Our Solution: Btrfs + Beesd Automatic Deduplication
``` ```
Windows Steam → NTFS Shared Library (real files, primary storage) Windows Steam → /steam/windows/ (btrfs via WinBtrfs driver)
↑ ↓
Linux Steam → ext4 Library (symlinks) ──┘ Linux Steam → /steam/linux/ (native btrfs)
↓
[beesd automatically deduplicates identical files]
``` ```
**Key Insight**: Make Linux the "smart" side that handles complexity, while Windows gets simple, reliable access to real files. **Key Insight**: Let each Steam installation work independently, rely on proven btrfs deduplication technology to eliminate duplicate storage automatically.
## Architecture Deep Dive ## Architecture Deep Dive
### Component 1: NTFS Shared Library ### Component 1: Shared Btrfs Filesystem
**Purpose**: Primary storage for all game files, designed for Windows Steam **Purpose**: Single high-performance filesystem for all game storage
- **Location**: `/mnt/shared-steam` (mounted NTFS partition) - **Location**: `/steam` (dedicated btrfs partition)
- **Ownership**: Windows Steam has native, unrestricted access - **Contents**:
- **Contents**: Real game files, installed/updated by Windows Steam - `/steam/windows/` - Windows Steam library
- `/steam/linux/` - Linux Steam library
- **Access**: Native on Linux, WinBtrfs driver on Windows
**Why this satisfies our goals**: **Why this satisfies our goals**:
- ✅ **Windows stability**: Native NTFS access, no drivers or special handling - ✅ **Automatic deduplication**: Beesd handles duplicate elimination transparently
- ✅ **Performance**: No filesystem translation layer for Windows - ✅ **No maintenance**: Zero custom scripts or symlink management
- ✅ **Reliability**: Windows Steam operates in its natural environment - ✅ **Independent operation**: Each Steam installation works normally
- ✅ **Performance**: Optimized mount options for gaming workloads
### Component 2: Linux Native Library ### Component 2: Beesd Deduplication Service
**Purpose**: Linux Steam's working directory with intelligent file management **Purpose**: Automatic background deduplication of game files
- **Location**: `/home/user/.local/share/Steam/steamapps` (ext4/btrfs)
- **Contents**: Symlinks to shared library + Linux-specific metadata
- **Management**: Automated script maintains symlink consistency
**Why this satisfies our goals**: **Core Functionality**:
- ✅ **Eliminate storage waste**: Symlinks use negligible space - Continuously scans `/steam` for duplicate blocks
- ✅ **Linux performance**: Native filesystem for Steam client data - Automatically deduplicates identical files between `/steam/windows/` and `/steam/linux/`
- ✅ **Graceful degradation**: Linux games still work if shared library fails - Operates transparently - games never know deduplication is happening
- Handles common duplicates: DirectX runtimes, Visual C++ redistributables, game engines, shared assets
### Component 3: Intelligent Deduplication Script **Expected Efficiency**:
- **40-70% storage savings** for typical game libraries
- **Common targets**: Unity/Unreal engine files, shared libraries, identical texture assets
- **Real-time operation**: New duplicates eliminated automatically
**Purpose**: Automatically manage game installations and eliminate duplicates ### Component 3: WinBtrfs Driver Integration
**Core Logic**: **Purpose**: Provide Windows with native btrfs read/write access
```bash
# Scenario 1: New Windows install
if [[ -d "$shared_path" && ! -e "$linux_path" ]]; then
ln -s "$shared_path" "$linux_path" # Link to Windows install
fi
# Scenario 2: Linux install exists, no Windows version
if [[ -d "$linux_path" && ! -L "$linux_path" && ! -d "$shared_path" ]]; then
mv "$linux_path" "$shared_path" # Move to shared storage
ln -s "$shared_path" "$linux_path" # Replace with symlink
fi
# Scenario 3: Both exist (duplicate install)
if [[ -d "$linux_path" && ! -L "$linux_path" && -d "$shared_path" ]]; then
# Intelligent merge: keep larger/newer, discard duplicate
deduplicate_installation "$game_name"
fi
```
**Why this satisfies our goals**:
- ✅ **Minimize maintenance**: Fully automated, runs on boot/schedule
- ✅ **Eliminate waste**: Automatically consolidates duplicate installs
- ✅ **Seamless access**: New games appear on both OSes transparently
### Component 4: NTFS Mount Configuration
**Purpose**: Secure, performant NTFS access with proper permissions
```nix ```nix
fileSystems."/mnt/shared-steam" = { # No special configuration needed - standard btrfs mount
device = "/dev/disk/by-uuid/YOUR-NTFS-UUID"; fileSystems."/steam" = {
fsType = "ntfs-3g"; device = "/dev/disk/by-uuid/YOUR-BTRFS-UUID";
fsType = "btrfs";
options = [ options = [
"uid=1000" # Your user owns all files "noatime" # Don't update access times - major gaming performance boost
"gid=100" # Users group "ssd" # SSD optimizations
"umask=022" # Secure but usable permissions
"dmask=022" # Directory permissions: 755
"fmask=133" # File permissions: 644 + executable
"windows_names" # Handle Windows naming conventions
"big_writes" # Performance optimization
"user_xattr" # Extended attributes for compatibility
]; ];
}; };
``` ```
**Why this satisfies our goals**: **Windows Requirements**:
- ✅ **Eliminate permission issues**: Consistent ownership and permissions - Install WinBtrfs driver (https://github.com/maharmstone/btrfs)
- ✅ **Performance**: Optimized mount options for gaming workloads - Configure Steam library path to point to `/steam/windows/`
- ✅ **Reliability**: Stable, well-tested NTFS-3G configuration - No special configuration needed - works like any other drive
### Component 4: Beesd Configuration
**Purpose**: Optimized deduplication for gaming workloads
```nix
services.beesd.filesystems = {
steam = {
spec = "/steam";
hashTableSizeMB = 512; # Sized for ~4TB of game data (128MB per TB)
verbosity = "err"; # Only show actual problems
};
};
```
**Hash Table Storage**:
- Stored as file: `/steam/.beeshash`
- 512MB handles up to 4TB of game data efficiently (beesd recommends 128MB per TB)
- Loaded into RAM on-demand, not all resident simultaneously
- Smaller hash table = better performance and less disk usage
## Partition Strategy
### 4-Partition Architecture
1. **Windows Boot Partition** (NTFS) - *Existing*
- Standard Windows system drive
- Unchanged from current setup
2. **Linux Boot Partition** (btrfs/ext4) - *New*
- NixOS system installation
- Standard Linux root filesystem
3. **Shared Steam Library** (btrfs) - *New*
- `/steam/windows/` - Windows Steam library
- `/steam/linux/` - Linux Steam library
- Automatic beesd deduplication
4. **Legacy Windows Steam** (NTFS) - *Existing, Optional*
- Keep for anti-cheat games that may not work on btrfs
- Can be eliminated if all games work on btrfs
- Provides fallback option during testing
### Multi-User Considerations
**Current Setup**: 3 Windows users sharing Steam libraries
**Migration Strategy**:
- Other Windows users continue using existing NTFS library
- Primary user (johno) experiments with btrfs library
- Easy rollback: point Steam back to NTFS if issues arise
- Gradual migration as confidence builds
## Implementation Strategy ## Implementation Strategy
### Phase 1: Base Setup ### Phase 1: Base Setup
1. **Partition Management**: 1. **Partition Creation**:
- Identify/create NTFS partition for shared library - Create new btrfs partition for `/steam` (recommend 2TB+ for modern libraries)
- Ensure sufficient space (recommend 1TB+ for modern libraries) - Install NixOS on separate Linux boot partition
- Keep existing Windows partitions untouched
2. **NixOS Configuration**: 2. **NixOS Configuration**:
- Add NTFS mount with optimized options ```nix
- Enable Steam with proper permissions fileSystems."/steam" = {
- Configure automatic script execution device = "/dev/disk/by-uuid/YOUR-BTRFS-UUID";
fsType = "btrfs";
options = [ "noatime" "ssd" ];
};
3. **Initial Testing**: services.beesd.filesystems = {
- Install test game on Windows steam = {
- Verify Linux can access via symlink spec = "/steam";
- Confirm updates work from both sides hashTableSizeMB = 512; # 128MB per TB recommended
verbosity = "err";
};
};
```
### Phase 2: Script Development 3. **Windows Setup**:
- Install WinBtrfs driver
- Add Steam library pointing to `/steam/windows/`
- Test with a few small games initially
1. **Core Deduplication Logic**: ### Phase 2: Gradual Migration
- Game discovery and inventory
- Symlink management
- Conflict resolution
2. **Safety Features**: 1. **Safe Game Testing**:
- Permission verification - Start with single-player games
- Backup creation before major operations - Test save game compatibility
- Rollback capability - Verify performance matches NTFS installation
3. **Automation Integration**: 2. **Anti-Cheat Evaluation**:
- SystemD service for boot-time execution - Test multiplayer games progressively
- Optional inotify-based real-time updates - Document which games work on btrfs vs require NTFS
- Keep problematic games on legacy NTFS partition
3. **Deduplication Verification**:
- Monitor beesd logs for successful deduplication
- Use `btrfs filesystem du /steam` to verify space savings
- Benchmark game loading times vs separate installations
### Phase 3: Optimization ### Phase 3: Optimization
1. **Performance Tuning**: 1. **Performance Tuning**:
- I/O scheduler optimization for gaming workloads - Monitor btrfs performance under gaming workloads
- NTFS mount parameter fine-tuning - Adjust mount options if needed
- Script performance optimization - Optimize beesd parameters based on actual usage
2. **Robustness Improvements**: 2. **Monitoring Setup**:
- Error handling and recovery - SystemD service monitoring for beesd
- Logging and monitoring - Disk space alerts for `/steam` partition
- Edge case handling - Basic health checks for WinBtrfs stability
## Trade-offs and Considerations ## Trade-offs and Considerations
### Advantages ### Advantages
**Storage Efficiency**: **Storage Efficiency**:
- ✅ Single copy of each game (terabytes saved) - ✅ 40-70% storage savings through automatic deduplication
- ✅ Symlinks use negligible space
- ✅ No artificial storage constraints
**Maintenance**:
- ✅ Fully automated after initial setup
- ✅ Self-healing: script fixes broken symlinks
- ✅ No manual intervention required - ✅ No manual intervention required
- ✅ Works with any game installation method
**Compatibility**: **Simplicity**:
- ✅ Windows Steam operates normally - ✅ No custom scripts to maintain
- ✅ Linux Steam gets full game access - ✅ No symlink complexity
- ✅ Updates from either OS benefit both - ✅ Standard Steam library management on both OSes
**Reliability**:
- ✅ Each Steam installation completely independent
- ✅ Btrfs and beesd are mature, proven technologies
- ✅ Graceful degradation if deduplication fails
**Performance**:
- ✅ Optimized mount options for gaming
- ✅ No compression overhead
- ✅ Native filesystem performance on both OSes
### Limitations ### Limitations
**Complexity**: **Windows Dependencies**:
- ⚠️ More complex than separate libraries - ⚠️ Requires WinBtrfs third-party driver
- ⚠️ Requires custom script maintenance - ⚠️ Driver updates needed with major Windows releases
- ⚠️ NixOS-specific configuration - ⚠️ Potential compatibility issues with some anti-cheat systems
**Dependencies**: **Complexity Trade-offs**:
- ⚠️ Relies on NTFS-3G stability - ⚠️ More partitions to manage than single-filesystem approach
- ⚠️ Script must run reliably - ⚠️ Beesd adds background CPU/disk usage (minimal but present)
- ⚠️ NTFS partition must remain healthy - ⚠️ Hash table requires disk space (512MB for large libraries)
**Edge Cases**: **Platform Compatibility**:
- ⚠️ Some games may have platform-specific files - ⚠️ Some games may prefer NTFS for maximum compatibility
- ⚠️ Very large libraries may stress script performance - ⚠️ Anti-cheat systems may flag non-NTFS installations
- ⚠️ Symlink chains could confuse some games - ⚠️ WinBtrfs stability depends on third-party development
### Risk Mitigation ### Risk Mitigation
**Backup Strategy**: **Backup Strategy**:
- Regular snapshots of shared library - Regular btrfs snapshots of `/steam` partition
- Script creates backups before major operations - Keep legacy NTFS Steam library as fallback
- Steam's built-in backup/restore as fallback - Steam's built-in backup/restore for critical games
**Fallback Options**: **Fallback Options**:
- Linux can install games locally if shared library fails - Easy to revert games to NTFS library if needed
- Windows operates independently - Linux can install games locally if `/steam` partition fails
- Script can be disabled without breaking either Steam installation - Independent operation means failure in one OS doesn't affect the other
**Monitoring**: **Monitoring**:
- Log all script operations - SystemD service status for beesd
- SystemD service status monitoring - Disk space monitoring for early warning
- Disk space monitoring for both partitions - Game launch testing after major updates
## Expected Outcomes ## Expected Outcomes
### Immediate Benefits ### Immediate Benefits
- **50-80% storage savings** (typical for large game libraries) - **40-70% storage savings** for typical game libraries
- **Zero maintenance** after initial setup - **Zero maintenance** after initial setup
- **Identical game performance** on both platforms - **Identical performance** to native installations
- **Future-proof** architecture using standard technologies
### Long-term Benefits ### Long-term Benefits
- **Simplified game management**: Install once, play anywhere - **Automatic optimization**: New games deduplicated without intervention
- **Faster OS switching**: No need to reinstall games when switching primary OS - **Simplified management**: One shared library instead of separate installations
- **Future extensibility**: Foundation for more advanced features (save sync, etc.) - **Technology leverage**: Benefits from ongoing btrfs and beesd improvements
### Success Metrics ### Success Metrics
- ✅ Games launch successfully from both OSes - ✅ Games launch successfully from both OSes
- ✅ Updates from either OS work correctly - ✅ Save games work correctly on both platforms
- ✅ No manual intervention required for normal operation - ✅ Updates from either OS don't break the other
- ✅ Storage usage comparable to single-OS setup - ✅ Storage usage 40-70% less than separate libraries
- ✅ No performance degradation compared to native installs - ✅ No performance degradation vs native installations
- ✅ Anti-cheat compatibility acceptable for target games
### Monitoring and Maintenance
**Automated Monitoring**:
```nix
# Add to NixOS configuration for basic monitoring
systemd.services.steam-health-check = {
description = "Check Steam partition and beesd health";
serviceConfig = {
Type = "oneshot";
ExecStart = pkgs.writeScript "steam-health" ''
#!/bin/bash
# Check beesd service status
systemctl is-active beesd-steam >/dev/null || echo "WARN: beesd not running"
# Check disk space
USAGE=$(df /steam | tail -1 | awk '{print $5}' | sed 's/%//')
[ "$USAGE" -gt 90 ] && echo "WARN: Steam partition >90% full"
# Verify both directories exist
[ ! -d "/steam/windows" ] && echo "ERROR: Windows Steam directory missing"
[ ! -d "/steam/linux" ] && echo "ERROR: Linux Steam directory missing"
'';
};
};
systemd.timers.steam-health-check = {
wantedBy = [ "timers.target" ];
timerConfig = {
OnCalendar = "daily";
Persistent = true;
};
};
```
## Implementation Files Needed ## Implementation Files Needed
1. **NixOS Configuration Module** (`steam-dual-boot.nix`) 1. **NixOS Configuration Addition** - Add to existing machine config
2. **Deduplication Script** (`steam-dedupe.sh`) 2. **WinBtrfs Installation Guide** - Windows setup instructions
3. **SystemD Service Definition** (`steam-linker.service`) 3. **Migration Checklist** - Step-by-step game migration process
4. **Installation Instructions** (`SETUP.md`) 4. **Troubleshooting Guide** - Common issues and solutions
5. **Troubleshooting Guide** (`TROUBLESHOOTING.md`)
This architecture provides a robust, maintainable solution that maximizes storage efficiency while preserving the stability and performance requirements of both operating systems. This architecture provides maximum storage efficiency with minimal complexity, leveraging proven technologies instead of custom solutions. The automatic nature of btrfs deduplication eliminates the maintenance overhead of complex symlink management while providing excellent storage savings.