Nothing to write /home about#

Sun 02 Apr 2023 05:10:43 PM UTC

This week in Liminix: a change in the plan. Much to think about, nothing to show

The original plan

In this phase we revisit this decision, and figure out how to gain the advantages of the Nixos options/type system while still allowing multiple instances of the same module

I spent a day or so thinking about this. The two fundamental tools for abstraction in Liminix are the module and the service.

• Liminix modules use the Nixos module abstraction (though not the actual modules themselves). A module can change global config - for example, to add users, groups etc, or add kernel symbols. Modules get their configuration by looking at some subtree of the big configuration attrset, and can define what "well-formed" looks like for the attribute values in that subtree.

• Services, on the other hand, are derivations that create s6 service directories. There is not a lot of type checking in service parameters, but what they do allow is having multiple instances of the same thing. A prime example of when you'd want this is on a device like the GL-AR750 that has two wireless radios (for 2.4GHz and 5GHz) that must each have their own hostap service. The config for each service is different, so it would be quite unwieldy to represent this using a single configuration object?

  (Yes, we could do config.network.wireless.wlan0 = {...} and config.network.wireless.wlan1 = {...} - for this application we can anticipate we might need more than one daemon. But do we do that for every module that might ever possibly need multiple services just in case, and how do we guess what they're going to be? Someone somewhere is going to want to run two http daemons, or four different ssh daemons on different ports, or something even weirder. DHCP clients on two hardware network interfaces and a VPN device that only exists while some other service is running and a named pipe connected to a Python script using scapy. Who knows? We may not even know what the config keys are for some of these potential use cases)

With that in mind, I've decided to defer the redesign of modules until later down the line when I also need to do some serious thinking about services. There's no point considering one without the other.

Writable filesystem

So onto the next part of the plan. As of March 2023 the Liminix filesystem is a “squashfs” image, meaning that the filesystem is generated at build time and read-only at runtime. Any updates or reconfiguration must be made by producing and flashing an entire new image, which is fine if you're making a big change anyway but not very convenient if you just wanted to install netcat or change the wifi channel.

I don't have anything here in commitable form yet, but I note the following as bullet points:

• our devices use "raw flash", so we should be using filesystems designed for raw flash (like jffs2 or ubifs) and not traditional block-device filesystems like ext[234]. This is to make sure that writes to the file system are wear-levelled (distributed across the device and not all done at the same spot).

• there are two kinds of raw flash: "nor flash" devices are usually smaller and are guaranteed (sic) to be free of bad blocks, whereas "nand flash" devices are bigger, but may develop bad blocks over time (just like hard disks did in days of yore) that the driver/filesystem is required to deal with. Therefore we'll be using ubifs for nand flash because it's based on UBI that tracks erase counts, and jffs2 for nor flash because we don't need that behaviour there, and because it turns out that ubifs has a minimum device/volume size which our nor flash devices are too small to qualify for.

• as followers of the gospel of Graham C we'd like to not make the entire storage device writable and end up with random state accruing on it. We expect that updates will be performed by building new packages on a separate build system and using nix-copy-closure to deploy, and that we should be able to generate any/everything in /etc, /var etc at boot time based on what's in the store and what's available from the (yet unimplemented) "secrets" provider.

• OpenWrt have a feature where they combine squashfs and jffs2 using a filesystem overlay - the idea being that because squashfs compresses better, they can ship the initial image as squashfs and then any post-install changes are made to the jffs2 overlay.

  This is neat, but I am not presently planning to do the same in Liminix - or at least until I have numbers to demonstrate it's useful. My thinking is that whereas an OpenWrt update to libc might just overwrite some files in /lib and all the existing binaries in the squashfs now find the new libc in jffs2, in a Nix system we don't have that. Upgrading libc means new builds of all the packages using it - because they previously depended on /nix/store/01234567-libc and now they depend on /nix/store/89abcdef-libc. If the old binaries were in jffs2 they could be overwritten and the space reused, but if they were in the immutable squashfs then that's now so much wasted space. </handwaving>

  Anyway, we'll see. It's less complicated to not do it, so that will be my default position.

Where I am right now, it looks like this means we'll be using initramfs to boot the system and then have an early userland that mounts the store from jffs2/ubifs and runs a script to recreate /etc and friends based on what's in it.

It ought to work, anyway. Next week maybe I can talk about it in the past tense and not the future conditional.

Writes of way#

Mon 10 Apr 2023 10:11:41 PM UTC

It ought to work, anyway. Next week maybe I can talk about it in the past tense and not the future conditional.

This update is a day late and a few hundred words short, but we're in a state of "it appears to work":

testbed login: root
Password:
login[326]: root login on 'console'


BusyBox v1.35.0 () built-in shell (ash)

# df
Filesystem           1024-blocks    Used Available Use% Mounted on
mtd5                     13184     12664       520  96% /persist
mtd5                     13184     12664       520  96% /nix
dev                      60156         0     60156   0% /dev
tmpfs                    60848       856     59992   1% /run
# ls /
bin      etc      lib      persist  run      sys
dev      home     nix      proc     srv
# touch /persist/HELLO
# ls -l /persist/HELLO
-rw-r--r--    1         0 Jan  1 05:43 /persist/HELLO

How did we get here?

• make qemu boot from an MTD instead of a virtio block device. This is a closer match with hardware characteristics, and in fact jffs2 won't run on a block device.
• add support for initramfs, consisting of a shell script /init and a very stripped-down busybox that can run it
• take the existing config.filesystem data that we're using to generate pseudofiles for squashfs, and use it to generate a shell script that runs from the initramfs and populates all the FHS-ish directories (/etc/, /var, /home &c). The script lives in the persistent filesystem, so that if we need to change config.filesystem then the change will take effect when we reboot and we don't have to rebuild the initramfs as well.
• fun with MTD erase block size

There is of course a bunch of cleanup to do, and some serious "what happened to all my storage space?" work - it might be a bit academic that I can write to the filesystem if there's less than half a megabyte of space left on it. Once we have some free space to play with we can try running nix-copy-closure

Some assembly required#

Sun 16 Apr 2023 08:18:04 PM UTC

tl;dr Premature optimization in the rootfs of my emu

Last week:

There is of course a bunch of cleanup to do, and some serious "what happened to all my storage space?" work

This week I've been playing with reducing the amount of storage used, by rewriting shell scripts in the initramfs as C programs. Nothing in the initramfs is accessible in the second stage environment, so once the system is booted it's just dead weight. Gien that a statically linked minimal busybox is around 200k, if we can get rid of it, that's a decent sized chunk of a 16MB flash.

It turns out that not only can we get rid of busybox, we can get rid of the entire C library. preinit.c uses the Nolibc minimal C-library replacement - a header file defining inline functions that implement common syscalls - instead of linking against Musl, and this results in an initramfs approximately 4k in size. So I'm pretty happy about that.

I'm less happy about having had to insert inline MIPS assembly at the top of main() to get it to work, mostly because of the strong possiblity I've got it wrong...

    asm("la $gp, _gp\nsw $gp,16($sp)");

Some slightly handwavey I-dont-fully-understand-this context here: $gp is the "global pointer" register on MIPS, which is used to make references to global variables use one instruction instead of two (this Nintendo 64 programming blog is the clearest explanation I've found). Why do we need this magic?

• the nolibc definition of __start doesn't include anything to set up $gp (compare with the equivalent musl code

• gcc is (sometimes) generating code to restore $gp from the stack after a function call returns - but nothing at function entry to save it on the stack in the first place. Which is odd, because it does save the register when I link against the regular C library.

• no combination of -G or -mgpopt flags to gcc seemed to change its behaviour here.

So first we load $gp from the _gp symbol using the la pseudoinstruction, and then we stick it into the offset on the stack where the compiler will expect to find it about twelve instructions later. If we disassemble the file we can see this in action: first the instructions we added, then it sets up the registers for a call to write (using gp-relative addresses), then when write returns it reloads $gp from an offset from $sp.

004000f0 <main>:
[...]
  40011c:       3c1c0042        lui     gp,0x42        # we added this
  400120:       279c8970        addiu   gp,gp,-30352   # and this
  400124:       afbc0010        sw      gp,16(sp)      # and this
  400128:       8f838018        lw      v1,-32744(gp)
  40012c:       00003025        move    a2,zero
  400130:       00661021        addu    v0,v1,a2
  400134:       80420000        lb      v0,0(v0)
  400138:       14400040        bnez    v0,40023c <main+0x14c>
  40013c:       8f90801c        lw      s0,-32740(gp)
  400140:       8f858018        lw      a1,-32744(gp)
  400144:       26100620        addiu   s0,s0,1568
  400148:       0200c825        move    t9,s0
  40014c:       04110134        bal     400620 <write>
  400150:       24040001        li      a0,1
  400154:       8fbc0010        lw      gp,16(sp)      # gcc wrote this
  400158:       00003825        move    a3,zero

It was a fun and entertaining voyage of learning new things, but also one for the unwritten Risk Register as I'm not happy I fully understand it. I suppose at least if it does fail then the mode of failure will be a quite obvious "doesn't boot".

Removing the shell from the initramfs also meant having to rewrite the activate script. This script lives in the actual root filesystem but is run in the initramfs context (no nix store, no shared libraries, minimal /dev, etc, and now no shell interpreter). For now this is just linked statically against musl and weighs in at around 70k, but really the same argument for using nolibc would apply just as much here - even though it's not part of the initramfs proper, that's still 70k that's not shared with anything else and we'll never use again after boot.

I have to be honest and admit that there are probably other parts of the system where I could make savings bigger than 200k. At least I hope there are - the jffs2 compression doesn't seem to be nearly as effective as squashfs, so we're paying quite a price for a writable filesystem. But there was still a certain amount of pride in making it almost-cost-neutral to add an initramfs.

Next week: adding multi-output derivations in the overlay. Looking at the generated filesystem I see that verious packages contain man pages, static libraries, random python files etc which are probably not necessary to the running of the system and could save us kB or even MB.

Erase and rewind#

Thu 27 Apr 2023 10:10:58 AM UTC

Late update this week, and it's little more than "hey, I'm still alive". I've been a bit under the weather and spending my discretionary computer time on playing GP Bikes instead of getting anything useful done.

What have we done? We reduced the rotuer configuration from over 16MB to about 11MB by

• fiddling with hostapd to build it without openssl

• adding hardeningDisable = ["all"] in the s6 derivation. Haven't investigated why this works yet, but without this flag, all my binaries - even "hello world" - are 70k or larger.

• removing the second copy(!) of busybox that the s6 init scripts were using

• replacing the ISC ntpd with chrony. While I'm sure it would have been possible to make ntp smaller, reports indicate that chrony has a much more "modern" security posture.

Also dicovered that jffs2 doesn't like being written onto a flash if you only erase the blocks that you're writing it to: it seems to want the rest of the flash to have been erased as well. That took a bit of figuring out.

More progress next week, I hope.