---
name: nixos-infect-raid
description: Convert a Hetzner dedicated server running Debian/Ubuntu with software RAID1 (mdadm) to NixOS using nixos-infect. Use when asked to install NixOS on a bare-metal server with RAID, or to recover/debug a failed nixos-infect attempt on RAID hardware. Only available to admin tenants.
allowed-tools: Bash(ssh:*), Bash(sudo:*), Read, Write, Edit
---

# NixOS-Infect on Hetzner Dedicated Servers with Software RAID

This skill covers converting a Hetzner bare-metal server from Debian/Ubuntu to NixOS when the server uses software RAID (mdadm). The standard `nixos-infect` script has **zero RAID support** and will produce an unbootable system without the fixes documented here.

## What nixos-infect gets wrong on RAID

1. **Imports `qemu-guest.nix`** — wrong for bare metal, adds VM-only modules
2. **GRUB on one drive only** — picks the first of `/dev/vda`, `/dev/sda`, `/dev/nvme0n1`; no redundancy
3. **No `boot.swraid.enable`** — initrd can't assemble md arrays → kernel panic at boot
4. **Missing RAID modules** — `md_mod` and `raid1` not in `boot.initrd.kernelModules`
5. **No `/boot` mount** — only generates the root filesystem entry
6. **Uses raw device paths** — `/dev/md2` instead of stable UUID references

## Pre-flight checklist

Before starting, gather this information from the target server:

```bash
# Check BIOS vs UEFI (critical — changes bootloader config)
ls /sys/firmware/efi 2>/dev/null && echo "UEFI" || echo "BIOS"

# RAID layout
cat /proc/mdstat

# Filesystem UUIDs (use these, not /dev/md* paths)
ls -la /dev/disk/by-uuid/

# Current mounts and fstab
mount | grep -E "md|raid"
cat /etc/fstab

# Block device layout
lsblk -f

# Network config (need to preserve this)
ip addr show | grep -E "inet |^[0-9]"
```

Also verify:
- SSH key access works (don't rely on password auth surviving the conversion)
- You have Hetzner Robot panel access for rescue system fallback
- RAID resync is complete (`cat /proc/mdstat` — all arrays should show `[UU]` with no resync in progress). Resync doesn't technically block the install, but it's safer to wait.

## Step-by-step procedure

### 1. Create the RAID overlay module

This file fixes everything nixos-infect gets wrong. Upload it to the target server:

```nix
# /root/raid-overlay.nix
{ config, lib, modulesPath, ... }:
{
  # Disable the VM profile that nixos-infect injects
  disabledModules = [ "${modulesPath}/profiles/qemu-guest.nix" ];

  # Enable software RAID assembly in initrd
  boot.swraid = {
    enable = true;
    mdadmConf = ''
      HOMEHOST <ignore>
    '';
  };

  # Hardware modules for bare metal
  boot.initrd.availableKernelModules = [ "nvme" "ahci" "sd_mod" "xhci_pci" ];

  # RAID modules — must be in kernelModules (force-loaded), not just availableKernelModules
  boot.initrd.kernelModules = [ "md_mod" "raid1" ];

  # CPU virtualization support
  boot.kernelModules = [ "kvm-amd" ];  # Use "kvm-intel" for Intel CPUs

  # BIOS: install GRUB MBR on ALL RAID member drives
  # Override nixos-infect's single-device setting
  boot.loader.grub.device = lib.mkForce "nodev";
  boot.loader.grub.devices = lib.mkForce [
    "/dev/nvme0n1"
    "/dev/nvme1n1"
    # Add all drives in the RAID array
  ];

  # Drop to shell on boot failure instead of hanging
  boot.kernelParams = [ "boot.shell_on_fail" ];
}
```

**For UEFI systems**, replace the GRUB block with:
```nix
  boot.loader.grub.device = lib.mkForce "nodev";
  boot.loader.grub.efiSupport = true;
  boot.loader.efi.canTouchEfiVariables = true;
```

**For Intel CPUs**, change `kvm-amd` to `kvm-intel`.

### 2. Run nixos-infect with NO_REBOOT

```bash
curl -fsSL https://raw.githubusercontent.com/elitak/nixos-infect/master/nixos-infect -o /root/nixos-infect
chmod +x /root/nixos-infect

NO_REBOOT=1 \
NIXOS_IMPORT=/root/raid-overlay.nix \
NIX_CHANNEL=nixos-24.11 \
bash -x /root/nixos-infect 2>&1
```

Key environment variables:
- `NO_REBOOT=1` — **critical**, prevents reboot before we fix configs
- `NIXOS_IMPORT` — path to our overlay, gets added to `imports` in generated config
- `NIX_CHANNEL` — NixOS version (e.g., `nixos-24.11`)

### 3. Handle the "duplicated devices in mirroredBoots" error

nixos-infect writes `boot.loader.grub.device = "/dev/nvme0n1"` in `configuration.nix`. Our overlay adds `boot.loader.grub.devices = ["/dev/nvme0n1" "/dev/nvme1n1"]`. NixOS merges these lists, causing `/dev/nvme0n1` to appear twice.

The fix is using `lib.mkForce "nodev"` on `.device` in the overlay (already included above). If you still hit this error, check that the overlay is being imported correctly.

### 4. Fix hardware-configuration.nix

Replace the generated file with correct RAID-aware entries using **UUIDs** (not `/dev/md*` paths, which can renumber across boots):

```nix
# /etc/nixos/hardware-configuration.nix
{ modulesPath, ... }:
{
  fileSystems."/" = {
    device = "/dev/disk/by-uuid/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX";
    fsType = "ext4";
  };

  fileSystems."/boot" = {
    device = "/dev/disk/by-uuid/YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY";
    fsType = "ext3";  # Check actual fs type with lsblk -f
  };

  swapDevices = [
    { device = "/dev/disk/by-uuid/ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ"; }
  ];
}
```

Get UUIDs from `ls -la /dev/disk/by-uuid/` or `blkid`.

### 5. Inline the overlay into configuration.nix

The overlay file at `/root/raid-overlay.nix` may get deleted during the infect process (it cleans up non-NixOS files). Move the overlay settings directly into `configuration.nix` and remove the import reference.

### 6. Build and verify

```bash
# Source nix if needed
source /root/.nix-profile/etc/profile.d/nix.sh 2>/dev/null

# Build the system (catches config errors before reboot)
nixos-rebuild boot

# Verify GRUB is on all drives (output should confirm each drive)
# The build output will show:
#   installing the GRUB 2 boot loader on /dev/nvme0n1...
#   installing the GRUB 2 boot loader on /dev/nvme1n1...
```

### 7. Reboot

```bash
reboot
```

**Boot takes ~90 seconds** on RAID systems due to md array assembly. Don't panic if SSH doesn't respond immediately.

If boot fails, use Hetzner Rescue System:
1. Hetzner Robot → Rescue tab → Activate Linux rescue
2. Robot → Reset tab → Hardware reset
3. SSH into rescue with provided credentials
4. Mount the NixOS root and fix config:
   ```bash
   mdadm --assemble --scan
   mount /dev/md2 /mnt
   mount /dev/md1 /mnt/boot
   # Edit /mnt/etc/nixos/configuration.nix
   # Then chroot and rebuild, or fix and reboot
   ```

## Typical Hetzner RAID1 disk layout

```
nvme0n1 / nvme1n1 (or sda / sdb)
├── p1 → md0 (swap)
├── p2 → md1 (/boot, ext3)
└── p3 → md2 (/, ext4)
```

## Post-install recommendations

- **Hostname**: Change from the Debian default (`Debian-bookworm-latest-amd64-base`) to something meaningful
- **stateVersion**: Set to match your NixOS channel (e.g., `"24.11"`)
- **SSH keys**: Verify your authorized_keys survived the conversion
- **Tailscale**: Install and join if using for admin access
- **Backups**: Set up borgmatic before doing anything else on the new system

## Reference: complete working configuration.nix for Hetzner RAID

```nix
{ config, lib, modulesPath, ... }:
{
  imports = [ ./hardware-configuration.nix ];

  services.logrotate.checkConfig = false;
  boot.tmp.cleanOnBoot = true;
  zramSwap.enable = false;
  networking.hostName = "my-server";
  services.openssh.enable = true;

  users.users.root.openssh.authorizedKeys.keys = [
    "ssh-ed25519 AAAA..."
  ];

  # Bare metal — not a VM
  disabledModules = [ "${modulesPath}/profiles/qemu-guest.nix" ];

  # Software RAID
  boot.swraid = {
    enable = true;
    mdadmConf = ''
      HOMEHOST <ignore>
    '';
  };

  boot.initrd.availableKernelModules = [ "nvme" "ahci" "sd_mod" "xhci_pci" ];
  boot.initrd.kernelModules = [ "md_mod" "raid1" ];
  boot.kernelModules = [ "kvm-amd" ];

  # GRUB on both drives for RAID1 redundancy
  boot.loader.grub = {
    enable = true;
    efiSupport = false;
    devices = [ "/dev/nvme0n1" "/dev/nvme1n1" ];
  };

  boot.kernelParams = [ "boot.shell_on_fail" ];
  system.stateVersion = "24.11";
}
```
