Side quest#

Sun 19 Mar 2023 01:59:50 PM UTC

This week: I made a video, then I went on a tangent.

The things I said last week that I was going to do this week were fixing a bug in the backup server, and adding NTP - but all the devices I need to do them to are deployed to "in the field" and I would rather not have to unplug them again and reattach the serial console wires to update them.

How can we make this better? After a bit of thought I have a new approach, which works in principle but needs a bit of polish and more testing. We use a combination of kexec and the watchdog timer to allow a new configuration that runs entirely in RAM to be booted from a running system. If the new configuration is doesn't work right, the watchdog will reboot the device - and because the new system hasn't been written to flash, then it'll boot to the previous configuration instead of bootlooping the new one.

The new configuration is built using outputs.kexecboot, which generates a vmlinux, a dtb and a squashfs that can be transferred to the device e.g. via ssh:

$ nix-build --show-trace -I liminix-config=./examples/arhcive.nix \
  --arg device "import ./devices/gl-ar750" -A outputs.kexecboot \
   &&  (tar chf - result | ssh root@192.168.8.185 tar -C /run -xvf -)

and then on the device we do

# kexec --map-file squashfs@104857600 --dtb dtb \
  --command-line "mtdparts=phram0:9945088(rootfs) phram.phram=phram0,104857600,9945088 memmap=9945088$104857600" \
  kernel

(the magic numbers in this command will be generated appropriately by tooling) to map the new root filesystem into a "phram" fake MTD device and then boot the new kernel.

This involved

• a patch to kexec-tools to add the --map-file option, which I hope to be able to upstream after cleaning it up. My use case (create an MTD for the new system) is probably quite niche, but the affordance in kexec-tools (map any file anywhere you like!) is pretty generic so perhaps there are other uses it could be put to as well.

• a small change to the kernel so that kexec provides the DTB address to the kernel being exec'ed instead of just copying it into a place where it might find it if MIPS_RAW_APPENDED_DTB is set.

Other fixes this week:

• add kernel config to make USB host work on gl-ar750
• make ssh key authentication work for root
• extract repeated code for applying openwrt kernel patches into a derivation

Next week ... the things we were originally going to do this week? I'd like also to set up a mailing list for the project, but am trying to find something more usable/searchable for the archives than Pipermail.