Previously, I left off the story with a working "bootloader" of some sort. It is a tiny Linux kernel, with an even tinier initramfs, packed on a Floppy disk. Once it loads it continues booting from the USB drive. But what should it boot from there?

Well, crafting this loader was fun and all, but it would be nice if I could just load up a proper distro from there, and not reinvent the wheel (or so I told to myself).

So I just unpacked the x86 build of Alpine Linux on the USB, set a few things up, and attempted to boot it. Well, it kind-of loaded, but sadly, most of the user-space binaries are compiled for i686... Running this on the i586 (The Pentium-MMX) is mostly... unlucky... binaries were crashing here and there attempting to execute "illegal instructions".

I had two paths ahead of me:

• Do a little research on what distro is still built for i586 and attempt to use that,
• or reinvent the wheel.

Well, obviously it was time to reinvent the wheel.

Coming up with Memes-OS

I don't think there is a clear definition of what's an "OS" or a "distro" is actually... ask 10 people and you'll get at least 11 different answers. I think we can somewhat agree, that a kernel plus some userland could be considered as such.

Looking at, what I had on my hand so far: I'm capable of compiling the Kernel and BusyBox (and kexec-tools) to run on this machine. BusyBox can basically be a userland, so I have everything to build some minimal OS thingy to continue, right?

In theory? Yes. In practice, the road was a bit bumpy however...

I actually never wanted to build a OS or distro, I just wanted to build the kernel and some user-space stuff, a friend of mine had to remind me, that what I'm doing is basically a new distribution.

Well, if we are up to building a new distro, then it will need a name! Coming up with one was easy: What does Memeloader load? Memes obviously. So we got Memes-OS.

From now on, I'll refer to the loader as "Memeloader", and the thing that it loads "Memes-OS". In the code they are just simply called "loader" and "target".

The Bigger Kernel

Memeloader's kernel had to be very tiny, so I started it off from tinyconfig. Initially I started of with tinyconfig here as well, but I had to change so many things, I instead switched to i386_defconfig and removed and added stuff to that.

For Memeloader I recorded all changes in a script with the help of scripts/config, this was okay because it only added a small number of things. I used the same approach here, but eventually, this list grew significantly, so maybe it would have worth to structure it a little bit better, but it worked so, I didn't bother.

In this bigger kernel, I had a lot more freedom on what to include, but I still wanted to keep it small-ish, because RAM is still scarce. I also had the idea of less features = faster system in my mind, so I turned off a LOT of things that are not needed for this project, but would be very well needed in any other scenario, like most of the security features....

I also added some things I felt the need for, or just thought it would be nice. This time, I went big, and added ext4 to the kernel (the ext4 code can actually mount ext2 partitions, so I'm still running off ext2, but now using a bit newer driver for it, at least).

I did not wanted to mess with kernel modules, and module loading (although BusyBox provides a modprobe of some sort), so I built a "static kernel", meaning all modules are compiled in to the Kernel, which is a single binary. Modules would make sense on a system, where you don't know in advance what features you will need. But I know exactly what I need. I'm building a whole system to run on this very specific laptop. So that's one problem less... (or so I thought)

Booting the "Second Stage"

I also figured, that for this second stage, we could skip the initramfs altogether, since the Kernel is capable of finding and mounting it's rootfs. I could just direct it to mount the USB drive as-is. However, I wasn't aware that the kernel has the most useless ways to specify where the rootfs lives (at least to us).

Basically I could use the usual /dev/sdXY pattern, but that could change during handoff between the two kernels. I could address the partitions... but remember, we don't have any partition table on the USB disk. Or I could use major/minor numbers to address it.

I ended up using this last one. I'm not sure how major/minor numbers work, but I assumed that it does not change during handover. I read the current major/minor numbers in the Memeloader script, and passed it to Memes-OS' kernel's commandline. The only problem is that after handoff, the Kernel still needs time to detect the USB drive, so by the time it tries to mount the filesystem it's not ready, so it fails to boot. The rootwait parameter would solve this, because it allows for the root device to show up later, and continue after it is detected. Sadly when using major/minor numbers as the root parameter, this didn't work (at least for me). I checked the source code but couldn't figure out the reason behind it. I just got VFS: Cannot open root device "8:0" or unknown-block(8,0): error -6 and then VFS: Unable to mount root fs on unknown-block(8,0). I assume that the code responsible for async waiting would return major/minor numbers. However, since we provided those exact numbers, it simply passes them trough without actually waiting.

Anyway, I had to fall back to rootdelay instead, which just a dumb "pause" before it mounts the root filesystem. It always waits the given time, and then attempts to mount the rootfs whether it's present or not. Interestingly it takes a lot of time (around 5 seconds) for the USB drive to show up. So I configured a generous 15 seconds here which is now always delaying the boot.

On some occasions the USB drive does not get detected even after this 15 seconds. I could not figure out the reason for that either. That is why I added that part of the script to Memeloader which un-mounts the USB stick, then shuts down the USB controller (well, actually it kicks it off from the PCI bus... not sure if that counts as proper shutdown). This seem to have made it a little more reliable, but occasionally it still fails.

Now, we got to the point where the bigger kernel starts up properly, mounts the USB drive, and attempts to run the init from it. Where to go from here?

What's an init Anyways?

Initially, I used /bin/ash as the init, as it was good enough. I had a shell, I could do things, and I was happy. I thought that's all what I'll need. Only to realize what was the meaning of that single warning I always tend to ignore:

/bin/ash: can't access tty; job control turned off

Among other things, this warning means, that CTRL+C is not going to work for sending interrupts to running programs. I had to realize this later in the experimentation, when I started a ping, and could not stop it. I had to reboot to stop pinging.

I was sure, that CTRL+C is a luxury that Memes-OS can not miss out on. So instead of specifying /bin/ash as the init, I turned to the proper looking /sbin/init binary, provided by Busybox (I think Alpine Linux does something similar, this is the binary that launches OpenRC for them).

Actually I put a tiny script in-between to prepare a few things, but that's just a minor detail.

This not only gave us job control and thus CTRL+C support, but also some other nice things, like multiple VTYs and VTY switching, configurable CTRL+ALT+DEL, ability to launch services, aaaand a login prompt. All configurable from /etc/inittab. (That means, we'll need to have /etc/)

Now that I had a login prompt, I know that I can't just log in using the root account. No. I needed something fancier. Like a regular user... but let's put this thread away for a little while now.

Filling up /etc/

The story telling is already gone a bit out of order, but now it will be intentional. It's very hard to give a nice structure to all the mess I did.

We already having /etc/ and the intention to build a more complete system. So let's give way to the intention and put stuff in there. We'll need a passwd, group and a shadow file to have users. (That also required enabling some crypto stuff in BusyBox but all hardware acceleration for it turned off, otherwise it wouldn't compile with our toolchain, well obviously.) I just made up some very minimal config (mostly by copying from Alpinelinx and my Debian PC). With those added, I could log in as a regular user! Awesome!

Let's just throw in a few more things. A profile file would be very nice to set some envvars like $PATH and $PS1 up. I actually came up with this script by myself, so I'm not sure if this is how you supposed to get $EUID.

I'll also need stuff like securetty and shells, for login and shell related things. I know for sure the later is used by chsh. If the first one is configured wrongly, You'll get a very confusing invalid password for 'root' on 'tty1' message when trying to log in as root.

Throw in a hosts file and a services to supposedly make network stuff happy. (Copying it from places again.)

And of course, the crown jewel of any good /etc/ a motd file! I always get nervous about that and end up with a rather lame one, and a promise that I'll make something nicer one day, which I never do... If anyone know how to make good small ASCII art, hit me up!

I added some bits to the build script to change passwords, and create a normal user, and that's kindof the point where this turned from a kernel and some userspace software to a distro of some sort... I should have added a os-release file...

One thing...

One thing still wasn't perfect. Veteran Linuxers might already figured it out, I needed a facepalm to realize it.

It's cool and all that I could log in as a regular user, but what if I want to change my password? I just use the passwd utility, but... wait... it does not work... hmm... let's just hop into a root account then with su - and... oh, this does not work either... OH RIGHT those binaries need SUID to work... which they does not have... let's just give them... oh right, this is busybox... everything is linked to the same binary. I think, it would be a bit unlucky to run all applets as root... maybe...

So I had to figure this out somehow. What would be the best way to have SUID with BusyBox? BusyBox has a feature to always drop SUID for applets that does not need it, to solve this exact scenario... but eh, I'm not a fan of this, adding extra code to solve what seems to be a problem that can be solved without adding extra code. Reading the help text for this feature though gives us a better idea. What if we use two BusyBox binaries? One with SUID and one without, and link the appropriate applets to each of them. Actually, while we are at it, we could build two separate binaries with only the applets that need or does not need SUID.

I wasn't really convinced of that solution either, so I looked at how Alpine Linux solved this. Turns out they do exactly that... they even have a separate package for that.

So that's what I did: I split Memes-OS' BusyBox build into two pieces, named the new one bbsuid (just how Alpine does), and built two separate binaries. I even split up their configuration to three pieces: one applet list for the SUID one, one for the other, and a common part, which does not enable any applets, only contains the generic configuration, so the two binaries will use the same crypto settings of passwd for example.

Now I had this all set up, I could finally change my password, or switch to a root account using su - (And also use the edgy version of ping).

One more thing...

As I might have already mentioned, I'm not great at compiling C programs with all this cross compilation magic. So adding new software was always a choice I thoroughly considered. Especially if those software would depend on other libraries, which must be compiled as well.

Speaking of libs, I couldn't really figure out that dynamic linking thing. So all the binaries I compiled throughout this project are statically linked. Therefore I did not needed a /lib/ folder in the rootfs (initially).

However I ran into issues later where programs just throw seemingly false error messages about files, permissions and connections. I usually have a pretty good instinct to reach for strace in those cases, but oh well... sadly BusyBox does not come with strace.

I thought it through at least three times and talked myself out of adding yet another software to the mix, but I couldn't help, my curiosity won. I needed to resolve those issues.

Thankfully, compiling strace ended up pretty easy, so my fear was for nothing. I just grabbed the source code, threw the magic envvars to it, and it compiled. No questions asked.

I don't want to spoil the software I had to debug, but it was throwing a Could not connect error. With strace I was able to figure out what the real problem was. I saw on the output that it was trying to access stuff in /tmp/... but it was getting ENOENT back. Well I can't blame it, we didn't even had /tmp/.

What is a real Linux distro without /tmp/, right? Let's add it to the build script quickly! On most modern distros, /tmp/ often mounted as a filesystem in RAM, to speed up these systems. Well, I'm not that concerned about speed here, but I am much more concerned about RAM usage. So I'll do this the retro way of putting /tmp/ on the disk (the USB drive in our case), and, remember the tiny script before starting init? I added a command in there to clean out /tmp/ on each boot. I don't think many software will need temp files here, but those that need should handle a little delay.

I've got to say strace overall is a great tool! Over the past it helped me untangle a lot of similar strange issues. Here it also helped me figuring out why telnetd didn't start, but let's not get too much ahead of ourselves again.

Getting on the Line

Memes-OS' kernel is now compiled with all the basic TCP/IP networking stack I'll need. I also have the network configuration and testing utilities from BusyBox. So everything is given to get online, right?

Oh yeah... a network interface...

Sadly the target machine does not have any modern network interface. I could, in theory, set up something like SLiRP over the serial port, but eh, that would make this reliant on another computer too much. I wanted this to be as self-contained as possible.

Then I realized, that I'm in the possession of a USB WiFi stick. I also dug up a USB HUB, and connected both the stick and the USB drive to the laptop. Much to my surprise, what worked before, didn't break, that's a good sign! I just needed to get the WiFi stick working!

To get it working, I needed to add all the necessary bits to the Kernel, for WLAN over USB to work, as well as the specific driver for this adapter. The chip in this thingy is a RTL8192EU, so I added the kernel driver for that single chip. Fun fact: this driver depends on CONFIG_LEDS_CLASS, this is how I realized that there is actually an LED on that dongle, which I didn't know before.

As with all similar devices, this one also needs a firmware to be loaded. So I downloaded the linux-firmware bundle, and extracted all blobs under rtlwifi to /lib/firmware (yup, added /lib/ only for this) on the rootfs. I have no idea, how firmware loading actually work, but seems like that's everything that is needed.

I booted up my contraption, plugged in the stick, and there it was! wlan0 appeared! Now what?

Other funny thing: The WiFi stick only works if I plug it in AFTER the OS booted up. That's because, as I mentioned previously, I built a static kernel. So the WiFi stick's driver present in the kernel before the rootfs is mounted, and it tries to initialize the device, but when it tries to load the firmware it fails, because the rootfs is not mounted yet. I decided to ignore this issue.

I had no idea, how to configure this WLAN interface. I looked trough BusyBox applets to see if there is anything to help, but no luck. Turns out "modern" WiFi adapters require a supplicant software to be running, such as wpa_supplicant, which will configure it up to the point that it can associate with an AP and keeps running in the background to do things needed for that. Well, we are compiling yet another software I guess...

Compiling wpa_supplicant proven to be a harder nut to crack. It had a nice, Kconfig like config file to configure it (similar to the Kernel's and BusyBox's but without menuconfig). To compile it, I applied the same practice I figured out for kexec and strace already by throwing the magic envvars to it.

AI Disclosure: I think it's time for this... I wanted to carry out this project without any LLM usage. I almost succeeded, but linker errors went above my head. wpa_supplicant did not wanted to play nicely, and threw a few hundreds of linker error at me which I couldn't really figure out. I copied a few lines into ChatGPT which helped me make more sense of it. After that I could come up with a configuration that finally built with this toolchain. Other than that, and an other similar linker error with BusyBox it helped decipher earlier, I have not used LLM help throughout this project. Though I ultimately failed.

The configuration that eventually built basically needed CONFIG_TLS=internal and CONFIG_INTERNAL_LIBTOMMATH=y. Normally wpa_supplicant would link against OpenSSL or something similar, but I didn't felt the chi to compile those as well, so I figured, roll your (or their) own crypto will be fine for now. LibTomMath is a mathematical library that is used by the internal TLS implementation... thankfully wpa_supplicant also includes a simplified version of that which we can just use. I however needed a single separate library called libnl. If I understood correctly, that is the library needed to interface with newer WLAN drivers that rely on CFG80211/MAC80211 which is what our Realtek driver uses here. Thankfully the usual process of throwing the envvars to it worked flawlessly, so I could build this lib and link wpa_supplicant against it.

After all that struggle, I had a freshly compiled, steaming hot wpa_supplicant binary, as well as wpa_cli and wpa_passphrase. Wonderful! (Actually it was wpa_cli that needed strace to debug and made me realize I missed /tmp/)

Added those to the rootfs and after some fiddling I had the WiFi stick associated to the AP just fine. Great success! Now we just need an IP address. I invoked udhcpc that is provided by BusyBox to please, obtain an IP address for me. Which it did! But something was off again...

udhcpc announced with great pleasure that it obtained an IP address... but did not configure that IP address on the interface... After a little research I found that it needs a "script" to do that, which it will invoke with specific parameters and expects it to do the magic.

I know Alpine uses udhcpc and the ip config utilities provided by BusyBox by default, so it must had such a script. Which it did! So I just stole that as well. Placed it under /etc/udhcpc/script and now, invoking udhcpc configures the IP address and the DNS server as well.

A quick ping to google.com and WE ARE ONLINE!!!!

---

I'm so happy, that I was finally able to complete Part 2 of this post. There is only one part left to tackle. Thank you a lot for sticking with the story! I grateful each one of you for reading through all this! I hope you could take away something for this weird adventure. I'll post the last part here on Cyberspace as well, and then I'll start working on adapting all this to my blog.