I'm still struggling to determine what kind of setup I want to use for my new home-server. I've watched countless Youtube video's about Proxmox, TrueNAS, TrueNAS in Proxmox, Debian in Proxmox and more and more alike. It all boils down to "use Proxmox as your hypervisor and use TrueNAS as your NAS" (either TrueNAS Core or TrueNAS Scale). I have one issue with this though; how do I get the ZFS datasets running from virtualized TrueNAS to other VM's? So far the answer seems to be SMB or NFS. There's one issue with that; speed.
In my current setup TrueNAS Core shares datasets with FreeBSD-jails (running apps like Plex, Sonar, IceCast) using null-mounts. You mount a dataset from the host into the jail and you're done. No hassle with NFS, SMB, iSCSI, just simple null-mounts. You can make them readonly which is nice, you generally have less issues with access-rights and most important of all; it's FAST. You're accessing the dataset directly and not using any layers in between.
I did run 2 VM's with BHyve on TrueNAS core, this is slow to begin with, and you can't use null-mounts, so accessing data from the host in a timely fashion is quite a struggle. Those who remember the days of FreeNAS (especially 9.x) know the pain with PHP-virtualbox, so BHyve looked like a blessing at the time ;)
I've come up with several scenario's which I'm currently investigating.
1. Proxmox Host, TrueNAS Core VM
Effectively moving my current server to a VM
I'll be installing TrueNAS Core 13 as VM running on Proxmox. All 3.5" disks (storage) will be mapped to the VM and TrueNAS will expose the datasets using NFS, AFP, SMB or iSCSI. There's one very big advantage to this setup: I'm able to transfer all my drives from my old server to the new one (even the SSD's running the jails) and restore the configuration from the old server in this new VM. All services will be up & running in no-time and to a certain degree I have the power-efficiency I was looking for. In time I'll be moving the jails in TrueNAS to VM's in Proxmox.
Another advantage; I'll keep using FreeBSD under the hood, albeit virtualized. I like FreeBSD.
The downside; I have 6 SATA ports on the motherboard available and I will need all of them to run the 4+2 disks from my old setup. So there's no room for an extra set of SSD's to run new VM's. I can solve this by adding a PCI-SATA controller (which I still have laying around) however; this will increase power-usage.
Another downside; FreeBSD (on which TrueNAS core runs) is not known as being a very power-efficient OS, so eventually I will need to replace TrueNAS Core with something else when I'm done moving all the jails to VM's.
2. Proxmox with TrueNAS Scale VM
I'll be installing TrueNAS Scale as VM running on Proxmox. My current dataset can be easily imported. I will have to rebuild all jails in VM's in Proxmox, datasets will be shared using NFS or SMB (no AFP support in Scale unfortunately).
I can use plugins (K8s) in Scale to run services, this would make it a bit easier to migrate all my stuff. I can even create VM's in a TrueNAS VM with KVM! I tried that and performance ain't that bad, not nearly as fast as a VM on Proxmox though. True VM-Inception!
But then again; TrueNAS Scale has KVM support, maybe not as incredible as Proxmox but good enough for me. Both are Debian Based, both are fairly energy efficient, so the real advantage of virtualizing TrueNAS is not really clear to me. The only thing I came up with; I can manage the TrueNAS VM (rebooting and such) easily from Proxmox.
3. Proxmox with Debian VM (and Docker!)
This solution was brought to my attention from one of the nerds from Met-Nerd-Om-Tafel-Slack. I'll be installing Debian as VM in Proxmox. Let Debian deal with ZFS and share the dataset. Debian will also run docker for services. Other VM's can be run from Proxmox (like PFSense or Home-Assistant) so if the Debian-with-docker instance needs to reboot other services won't be impacted (as long as they don't use the shares from the dataset).
This is a terminal only-solution (you could run a desktop environment from Debian though). I can manage that but since I've been using FreeNAS I did get used to managing my server with a web-interface. One big plus; this is very power-efficient.
4. Just TrueNAS Scale
Not virtualized, just TrueNAS Scale on bare metal. All the advantages from #2, but without overhead from Proxmox (though it's not clear how much this overhead really is). I can run K8s-templates/apps, Docker and Docker-compose, use KVM, have a nice NAS-solution and pretty web-interface.
I have not tried it yet but I did run into some networking issues when I ran TrueNAS on a VM, so those issues need to be ironed out before I'm satisfied with this choice. And i'm not that versed with k8s yet, but the UI from TrueNAS takes a lot of complexity away.
#5 Just TrueNAS Core
This would be by far the most easy way to migrate. Just move the SSD's, HDD's and boot-USB from old server to new one, boot it up and I'm good to go. But there's no fun in that and the power-efficiency of FreeBSD (or lack thereof) would become an issue for me. Oh; and the pretty bad virtualization support.. So, this is no-deal.
General thoughts and unanswered questions
There are some general issues I've been wondering and thinking about.
- I only have 32gb's of RAM. Will that be enough when I'm moving to more virtualization?
- The Fujitsu Board (D3644-B) has 2 M2-slots. One 80mm for SSD's and one 30 mm for Wifi. Can I use the latter M2 port for storage as well?
- The specs of the Fujitsu board say it supports up to 64Gb's of RAM, but I've seen reports it can also go up to 128Gb. Not sure if this is really the case though.
- I have a Intel 1gbit dual-nic PCI network card in my current server, should I move that to my new machine (the Fujitsu board has a Intel NIC as well). Could be of use when splitting up my network, but it might use more power.
- If I'd get a set of new SSD's for storing VM's, which ones would be suited for ZFS in mirror? I've read several posts on the TrueNAS forum about Samsung SSD's which fail after a few weeks due to too much writes. I always assumed Samsung SSD's were on of the best out there.
- Can I (and should I) forward the Intel GPU to a VM to run hardware encoding for Plex? I've seen video's of people doing that, but I'm uncertain if the onboard GPU can be forwarded to a VM.
- Does TrueNAS Scale support forwarding USB-devices to VM's? If not I won't be able to migrate Home-Assistant from my Raspberry PI4 to TrueNAS Scale.
I'll be doing some benchmarking and reviewing of each of the possible setups next days/weeks to get better understanding of performance impact and power-efficiency.. I'll reporting back here!