I’m getting a new circuit installed this week, so I can finally move my home lab off of my desk and into a small rack in the utility room. I haven’t decided most of the things I’ll be changing yet, but I wanted to show my current rough draft of my thoughts.

Home Lab 1.0 — Retrospective

Version 1 was a Supermicro SuperChassis (CSE-721TQ-250B) case with a mini-ITX Xeon-D board (Supermicro MBD-X10SDV-4C-TLN2F-O) with 32GB of RAM. It ran SmartOS (an illumos distribution) and used a combination of SmartOS and LX (Linux compatibility mode) containers. Debian was used only where appropriate. Alpine was the “default Linux”. Anything that would be better solved by SmartOS (basically all problems) used SmartOS.

A major upside of this configuration was that all the data at rest was in a hardware redundant ZFS pool. Over the years, a few hard drives went bad, but I was able to replace them without losing data. A container acted as a file server for my Macs, so I didn’t need to buy expensive Apple hardware to get a lot of storage. Originally, this machine had 6TB of storage, but I outgrew that and I’m currently at 32TB. Another upside was that it was just one machine with many contanerized services. It sat on the corner of my desk and was almost entirely silent. It ran file services, DNS, a Minecraft server, various git repos, automated backups of cloud services, Plex, and many ephemeral containers for screwing around with compilers and new software development tools.

The big issues I had with this system were related to KVM/QEMU or bhyve. KVM support came out first and worked out pretty well for me. I created a Windows Server 2008 VM and used that with Remote Desktop to do a lot of my side projects over the years. When I moved to bhyve, I followed a lot of tutorials, but never got it working correctly. bhyve isn’t as integrated, so you have to manually create a virtual network and attach it to the VM. This isn’t difficult to do, but it’s not obvious and it’s not well documented. Even after getting your VM set up exactly correctly with a manual bhyvectl command and a ZFS volume for storage, there’s some magic I’m missing to properly integrate this into the vmadm tooling. If I create a bhyve VM and replace the ZFS volume with a copy of what I previously got working, something doesn’t work correctly and the VM doesn’t boot. VNC never comes up and there are no errors in the console. I’m sure if I was more experienced with bhyve, I could figure out why the VNC service never comes up, but I’m just not able to figure it out.

Home Lab 2.0 — Planning

My current system is pretty close to my ideal, but there are some untried and experimental solutions I want to test out before I commit to a new system. My only real constraints are that I want to stay in the same energy ballpark. I have a solar system and I’m committed to keeping my lifestyle within the bounds of net metering.

Idea A: Oxide Computer created their own hyperscaler system that uses bhyve and illumos behind the scenes. I would like to use their API, but I want the backend to be either a cheap Xeon machine or a Turing Pi ARM64 machine. Maybe I could get both working. I need to follow the instructions and try to build an image that will boot. This isn’t impossible, so I will make an effort to get this working on my new home network. This is my ideal platform, but it technically doesn’t exist yet. The goal here is to do what Oxide is doing on a smaller scale. bhyve is higher overhead than I want, so I want to get this working with containers too. That might be beyond my capabilities, but I don’t know that yet.

Idea B: k3s is a lightweight Kubernetes distribution, which works on ARM64 and x86_64 hardware. This is probably the easiest technology to switch to, but it will require me to use Kubernetes tooling, which I can do, but maybe I don’t want to? (Explain this better) Things aren’t automatically good because they’re new. They have to earn being good by succeeding at my specific use case. I’m just running a few things in my house with solar power. I don’t need full Kubernetes. If I cut down to a subset of Kubernetes, have I lost what makes it great? I don’t know the answer yet.

Idea C: Proxmox is a VM solution, which is what I used before I switched to containers in 2008. I don’t really want to use VMs because they’re more overhead than containers. There are a lot of people who use Proxmox and VMs, so I want to make sure I investigate this option.

Idea D: SmartOS is an OS for datacenter nodes. It’s made to be netbooted, but you can run it as a live CD from a USB disk. This is how I’m using it. It would be really easy for me to keep this as my solution for my homelab, but I want to at least consider all the other options before I commit to this.