An operating system that I’ve been following for a while is NeptuneOS. The goal of this operating system is to implement a Windows NT ntdll.dll on top of the seL4 microkernel. MacOS and iOS both run on Mach, which was originally a pure microkernel, but traditional system calls were added later for performance reasons. The L4 microkernel is a next-gen successor to the Mach microkernel which solves a lot of the performance problems inherent in the previous generation. I don’t completely understand the differences, but Wikipedia has some explanations. SeL4 is a variant of the L4 microkernel which has been proven correct, which means that it won’t crash or deadlock on non-faulty hardware, assuming the specification of the OS is correct.
When NeptuneOS is launched, the seL4 kernel is loaded. Next, a file called ntos is loaded, which contains the “root task”, which owns and manages the rest of the tasks. Once the system is booted, applications are started. Each application links to a library called ntdll.dll. On Linux, applications communicate with the kernel via system calls, which tell the operating system what to do. On Windows, ntdll.dll exposes an API which doesn’t change from version to version of Windows. This is nicer, IMO, because Microsoft can redesign the system calls without breaking the API of the operating system. Neptune currently shows a command prompt interface, but at some point a Windows-like desktop interface will be possible.
It occurs to me that you could build something like DOS on seL4, not taking advantage of the microkernel’s affordances for subtasks. It’s not clear to me when you are meant to spawn a subtask and communicate via messaging or shared memory and when you should use a function call in the same task. In iOS 7, the software keyboard transitioned from an-process library to a separate tasks, which caused stuttering during typing on some older phones. I’m sure there are dozens of examples with similar tradeoffs inside the implementation of ntdll.dll. In the next few weeks, I plan to investigate this. Maybe I’ll find code which has been made into separate tasks which could have been function calls.
The remainder of this post will cover how to set up a Debian 12 (bookworm) system to build NeptuneOS from the sources on GitHub.
First, we install all the packages we’ll need.
apt install make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev curl git \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev \
libffi-dev liblzma-dev cmake jq ninja-build clang libxml2-utils \
cpio grub-common xorriso
Install pyenv, which will help manage multiple versions of Python.
curl -fsSL https://pyenv.run | bash
List the versions of Python which are available.
pyenv install --list
Install Python 3.10.0. You probably don’t need to use exactly the same version if you don’t want to.
pyenv install 3.10.0
Set Python 3.10.0 for the local session. This will take a long time, as it’s compiling Python from source. You can also use global to affect all logins.
pyenv local 3.10.0
Create a virtual environment for the NeptuneOS build and activate it.
pyenv virtualenv 3.10.0 neptune
pyenv activate neptune
Install all the Python dependencies.
pip install jinja2
pip install future
pip install ply
pip install setuptools
pip install six
pip install lxml
pip install pyyaml
Check out the repository.
git clone --recurse-submodules https://github.com/cl91/NeptuneOS.git
cd NeptuneOS
Check out the latest release.
git checkout tags/v0.2.0002
On Debian, a lot of the LLVM utilities are suffixed with their major version, 14. I decided to fix this by making symlinks in /opt/local/bin for each tool.
mkdir -p /opt/local/bin
cd /opt/local/bin
ln -s `which llvm-ar-14` llvm-ar
ln -s `which lld-link-14` lld-link
ln -s `which llvm-objcopy-14` llvm-objcopy
ln -s `which ld.lld-14` ld.lld
ln -s `which llvm-strip-14` llvm-strip
cd
cd NeptuneOS
Once everything is done, the build should succeed.
RC=`which llvm-rc-14` ./build.sh amd64 release
You’ll see this message when it’s finished building.
####################################
Build successful!
####################################
Once that’s done, you need to build a CD image with the following command
./mkiso.sh amd64 release
That’s not what’s happening today. Modern Apple’s intent is malicious. They want to capture more and more of your hard-earned money. The old Apple built Time Machine, which was designed to scale to your needs. You could buy a hard drive and plug it in to implement the basic support for incremental backups. If you had a fast network and AFP, you could use a network share instead. This could be way faster and could support redundant storage, so your data was safe from losing a disk. It was a “Choose Your Own Adventure” of how you wanted to implement the solution with Apple’s protocol.
New Apple built iCloud Backup, which was designed to scale to their needs of ever-increasing profit. They control how much it costs and how it’s implemented. As the cost of implementing this backup decreases, they’re pocketing the difference. Over time, they’re providing the same amount of storage for their users, but they’re making more and more profit.
Across much of Apple’s ecosystem are tools and products which are increasingly designed to extract your wealth, instead of the original goal of being a creative professional’s tool. In order for Apple to fix their software, they’re going to need to course correct upstream of the software. The leaders of the company must set goals of valuing users’ creative potential above profitability and those intents need to trickle down to the team leads, and then finally to the actual code itself.
]]>What I propose is that there are different places you can be in the programming universe. If you’re on C++ island, your truth is probably that lots of dynamic dispatch is bad and that types are good, but you can only take the type system so far before there are diminishing returns. The C++ community is full of people who all believe they’re in the same world, but they’re actually not. The community is fractured across a few core concepts being either good or bad. The truth is that most things contain both good and bad, but that’s another post. The divisive concepts are RAII, OOP, dynamic dispatch, exceptions, and more. There are probably thousands of distinct items you can argue about. One of my pet peeves is that C++ header files usually contain implementation details. Yuck. My preferred “C++” is using as little of these things as possible. A decade ago, Mike Acton gave my favorite talk ever at CppCon. He proposes that you mostly use C in C++ files. This is exactly what I do when I write C++. There are people lined up after the talk to explain how wrong he is. Building efficient system code goes beyond one intergalactic system language. You can’t describe all good system code in C++, so C++ can’t be the center of it all. I’ll come back to this in a bit.
There’s another island. The Alan Kay style OOP island (or maybe Adele Goldberg island) using late binding and message passing to build reusable software components which can snap together at runtime like Lego bricks. The reason I call out Alan Kay specifically is that his Smalltalk language used message passing instead of function calls and vtables. When you solve problems in this style, you use class clusters to break down a problem into its sub-problems. You think of the solution to a problem the same way you think of how you’d coordinate team members. Each person has a role and they communicate back and forth about their tasks. The great thing about this style of programming is that it maps to your own problem solving intuition. You can divide up the work and break it up among separate classes which coordinate to produce an output. This style of programming is distinct from others, which is why it lives on its own island. You will benefit from this style if you have a lot of business logic rules which map to human behavior. An example of a bad fit, which is my opinion BTW, is Apple’s CoreData. It was designed to be Alan Kay OOP, but is actually not a great fit because the problem itself is to get values from a database. You can trivially outperform CoreData with C functions which get the data. You don’t actually need class clusters to solve this problem.
Objective-C and C++ developers don’t really get along because the high-level code in Objective-C is actually from Alan Kay OOP island. It zero-cost bridges to C, which lives on C++ island. The fact that you have to be good at both programming methodologies to build certain types of apps is why Objective-C-based apps could be so good, but it’s also why it’s difficult to find people who can do that type of work.
There are more than just the two islands. Functional Programming is its own dinstinct island. You’ve probably seen people from this island showing up in your community. They’re known to be polyglots and are quite friendly, in my experience. There are truths from that island which are beneficial, so make sure to add some of these ideas to your brain if you want to.
Back to C++. An important thing to note here is that C++ island has never been just C++. It’s really a system code island. With Rust gaining popularity and getting to be a capable replacement for C and C++, it’s important that we change the name of the island in our hearts and minds. The real goal of this island all along is best described by Abner Coimbre in this talk. In the video, he says that good software is about transforming inputs into outputs in the most efficient way. I think this is completely true for games and operating systems, but not for all software. Note that Abner only really believes that C++ island exists and that everyone else is misguided.
A goal of mine is to always be kind to people who are entering your community and try to share space and power with them. Try to make some room for others. Be kind to Rust people who want to share this intellectual space with you. The best of the D programming language has been folded into C++ over the last decade or so. The same thing will happen with Rust. C++ can get optional lifetime tracking in the future and will be better for it.
]]>Version 1 was a Supermicro SuperChassis (CSE-721TQ-250B) case with a mini-ITX Xeon-D board (Supermicro MBD-X10SDV-4C-TLN2F-O) with 32GB of RAM. It ran SmartOS (an illumos distribution) and used a combination of SmartOS and LX (Linux compatibility mode) containers. Debian was used only where appropriate. Alpine was the “default Linux”. Anything that would be better solved by SmartOS (basically all problems) used SmartOS.
A major upside of this configuration was that all the data at rest was in a hardware redundant ZFS pool. Over the years, a few hard drives went bad, but I was able to replace them without losing data. A container acted as a file server for my Macs, so I didn’t need to buy expensive Apple hardware to get a lot of storage. Originally, this machine had 6TB of storage, but I outgrew that and I’m currently at 32TB. Another upside was that it was just one machine with many contanerized services. It sat on the corner of my desk and was almost entirely silent. It ran file services, DNS, a Minecraft server, various git repos, automated backups of cloud services, Plex, and many ephemeral containers for screwing around with compilers and new software development tools.
The big issues I had with this system were related to KVM/QEMU or bhyve. KVM support came out first and worked out pretty well for me. I created a Windows Server 2008 VM and used that with Remote Desktop to do a lot of my side projects over the years. When I moved to bhyve, I followed a lot of tutorials, but never got it working correctly. bhyve isn’t as integrated, so you have to manually create a virtual network and attach it to the VM. This isn’t difficult to do, but it’s not obvious and it’s not well documented. Even after getting your VM set up exactly correctly with a manual bhyvectl command and a ZFS volume for storage, there’s some magic I’m missing to properly integrate this into the vmadm tooling. If I create a bhyve VM and replace the ZFS volume with a copy of what I previously got working, something doesn’t work correctly and the VM doesn’t boot. VNC never comes up and there are no errors in the console. I’m sure if I was more experienced with bhyve, I could figure out why the VNC service never comes up, but I’m just not able to figure it out.
My current system is pretty close to my ideal, but there are some untried and experimental solutions I want to test out before I commit to a new system. My only real constraints are that I want to stay in the same energy ballpark. I have a solar system and I’m committed to keeping my lifestyle within the bounds of net metering.
Idea A: Oxide Computer created their own hyperscaler system that uses bhyve and illumos behind the scenes. I would like to use their API, but I want the backend to be either a cheap Xeon machine or a Turing Pi ARM64 machine. Maybe I could get both working. I need to follow the instructions and try to build an image that will boot. This isn’t impossible, so I will make an effort to get this working on my new home network. This is my ideal platform, but it technically doesn’t exist yet. The goal here is to do what Oxide is doing on a smaller scale. bhyve is higher overhead than I want, so I want to get this working with containers too. That might be beyond my capabilities, but I don’t know that yet.
Idea B: k3s is a lightweight Kubernetes distribution, which works on ARM64 and x86_64 hardware. This is probably the easiest technology to switch to, but it will require me to use Kubernetes tooling, which I can do, but maybe I don’t want to? (Explain this better) Things aren’t automatically good because they’re new. They have to earn being good by succeeding at my specific use case. I’m just running a few things in my house with solar power. I don’t need full Kubernetes. If I cut down to a subset of Kubernetes, have I lost what makes it great? I don’t know the answer yet.
Idea C: Proxmox is a VM solution, which is what I used before I switched to containers in 2008. I don’t really want to use VMs because they’re more overhead than containers. There are a lot of people who use Proxmox and VMs, so I want to make sure I investigate this option.
Idea D: SmartOS is an OS for datacenter nodes. It’s made to be netbooted, but you can run it as a live CD from a USB disk. This is how I’m using it. It would be really easy for me to keep this as my solution for my homelab, but I want to at least consider all the other options before I commit to this.
]]>