How it started

I've been running a home-server (NAS) for ages. It started back in the early 2000's with a Mac Mini and a bunch of USB-harddisks, connected with a USB-hub. I used it as NAS, recorded tv-shows with it (with an Elgato Coax-tuner), served PHP pages with Apache. Fun times ;) I bought a "real" NAS several years later (not sure but I think it was a QNAP) and kept using the Mac Mini for services.

Around 2010 and a lot of failed USB-disks later I build my first full system; a Tower PC with Core2Duo, 8Gb of RAM and 6x2Tb + 2x500gb HDD's. At first I tried to use Ubuntu with soft-raid until a friend mentions ZFS. I quickly installed PCBSD (FreeBSD with a desktop environment) and created my first ZRAID-dataset (I used ZRAID2 which was a wise choice in hindsight). I installed all services I needed in FreeBSD-jails and all was fine and dandy. This one day I lost 2 drives at once, both were pretty old and I was very happy I chose ZRAID2 for my dataset. After a while I stumbled upon FreeNAS and fell in love with it. I decided to backup all my data and setup my home-server with FreeNAS. Again; fun times!

In 2016 I decided to build a new server with new drives. My focus was on a more energy efficient system, and I needed more storage. So I build a new system with an I3-6100, 16Gb of RAM and 4x3Tb in ZRAID1. I added an SSD as ZIL. I used FreeNAS again.

via GIPHY

Around 2019 I replaced the disks with 4Tb models (I dropped WD Green in favor for WD Reds) added 2 brand new SSD's in mirror (for running jails and VM's) and expanded RAM to 32Gb. Eventually I upgraded FreeNAS to TrueNAS.

This system is still running. Doing a lot of stuff; downloading, VPN Gateway, Reverse Proxy, IceCast streaming, Security Cam Recorder, Plex Media Server, hosting websites, GIT, VSCode Server, Home-automation. Officially the hardware is still good to go, however; the energy efficiency is not that good compared to modern hardware. It's never idle, but doing it's thing it uses about 115watt.. Which is quite a lot.

So; time for a new home-server

Hardware

I spend a good month researching energy efficient parts, from the motherboard, the best CPU to new drives. The forum on the dutch website Tweakers.net has a very long thread about energy efficient hardware and I talked a lot about it with my nerdfriends on Met-Nerd-Om-Tafel-Slack.

via GIPHY

This is what I came up with;

• Intel I3-8100
• Fujitsu D3644-B Motherboard
• 2x Corsair Valueselect DDR4-2400 memory (non-ecc, yeah, I know)
• be quiet! Pure Rock 2 black CPU Cooler
• Corsair RM550x power supply unit
• Cooler Master MasterBox NR600 Midi Tower
• Gigabyte 128Gb PCI-E M2 SSD

I'll be reusing my 4x4Tb WD Reds and I'm planning on buying 2 Sata-SSD's. Eventually I'll add 2x16Gb of RAM to get to 64Gb. I expect the WD Reds to keep up for another 2 years after which they will be replaced by either 2x16Tb or a bunch of 2,5'' disks (which are more energy efficient).

After receiving all parts I went to work and in 2 hours I had my new system up & running. After some small tweaks in the BIOS (mainly enabling P-states) and installing Debian 11 I checked my power-meter; 7 watts idle!! now that's a difference! Ok, the system was running without the powerhungry 3.5" disks, but I'd looked promising already.

Next; determining what OS to use.

Software - Analysis Paralysis Awaits!

Kudos to Saber Karmous for the term Analysis Paralysis

via GIPHY

This is going to be a tough one. I have a list of must-haves and nice-to-haves and I don't want to have to start over because I'm not happy with the OS.

The list Must-haves

• Mature ZFS support
• Able to run docker containers
• Able to install apps from either templates (jails/k8s) or docker-compose
• Able to map USB-devices to VM's or container (in order to replace my Pi4 with Home-Assistant)
• Able to run VM's
• Assign a LAN-IP to each of the VM's or containers
• Energy-efficient (doh)
• Be extremely quiet
• Reuse my current ZFS dataset without moving data to a temporary set of disks
• Easy to restore in case of emergency
• Have decent throughput
• Run all the stuff I use already (preferably more :p )

The list of wanna-haves

• Able to assign VM's or containers to specific VLAN's (though I still have no idea how I got this working on my current setup)
• Capable of running OBS from a Linux VM and transcode 2 live-streams at once
• Replace my current fiber-router (OPNsense)

The contenders

via GIPHY

There's a lot out there. But this is my shortlist

• TrueNAS Core. Trusty TrueNas. I know my way around this OS, it's incredibly stable and secure. Does most of the things I want it to do, but there is one big downside; it does not run everything I throw at it. I've had several moments past few years in which I simply just could not get something to run on FreeBSD because there was no FreeBSD package for it. Yeah, you can use BHyve to start a VM, but it's quite cumbersome to configure and it's not fast.
• TrueNAS Scale. New kid on the block. Looks very promising. Based on Debian 11, with mature ZFS support, docker/k8s out of the box, virtualization with KVM. Looks like a perfect solution. And it probably is. I've tested it on my laptop in VirtualBox and I was impressed. I have not been able to assign LAN ip's to containers or VM's but I did not spend a lot of time on that yet, I'll probably get that working as well. This one I need to check out.
• Proxmox. Looks promising. I looked at Proxmox years ago, did not like it back then, but I saw a lot of video's about it and I think I should give it another chance. Also to be checked out!
• UnRAID. Has a lot of fans, I'm not one of them. The way UnRAID handles disks seems very efficient and safe, but I want ZFS, I like ZFS, I know ZFS and ZFS saved me several times before. There are plugins for UnRAID to have it support ZFS, but it feels a bit wonky to me. It's on the list, but I'm not sure if I really want to try it.
• Plain old Debian with ZFS. I can install another Debian VM on top of it with docker containers, for easy rebooting. No shiny GUI, just plain ol' terminal. But it seems flexible and possible incredibly fast. Must try this as well.
• Finally; Any combination of one of the solutions mentioned above.

Say what? Yeah, I came across this video on youtube where this guy has a Proxmox-host, installed TrueNAS Scale as a VM, pass along a hardware-controller with a bunch of harddrives to TrueNAS Scale and setup a ZFS dataset in the TrueNAS VM, returning the disks to Proxmox using SMB-shares. Proxmox was now able to run VM's from those SMB-shares. I was watching this and first thing that came to mind; what's wrong with this guy? Proxmox has ZFS support, why would you want to run ZFS from a VM? And why would you want to run VM's from a SMB-share???

I have questions

via GIPHY

I mentioned this TrueNAS as VM in Proxmox in Slack to my nerdfriends. The first reaction I got was; yeah, I have my server running exactly like in that video. Aight, this is a thing?

So, I'll be installing Proxmox, I wanted to take a good look at it anyhow. Time for some benchmarking! In a few days I'll report back here!

Subscribe to be updated when I report my findings!

via GIPHY