NixWRT - 5.4 branch#

Wed, 07 Oct 2020 23:08:15 +0000

I am starting this entry on Sunday evening, which means it might be finished in time for my once-traditional Tuesday blog post. No promises though.

Long story short: NixWRT has now switched to using Linux 5.4.64 as the base kernel version, and to using modules based on Linux 5.9 for wireless devices/802.11 code. It works for ramips and for ath79 targets: ar71xx has been removed.

Long story, unshortened: I have given up on trying to build monolithic kernels. Now I build the linux 5.4 kernel with support for most of the stuff that doesn't change a lot, then build wireless drivers and the wireless protocol stuff (lib80211, mac80211) using code from the linux-backports project. This is all made possible by the Linux Backports Project

Linux-backports is quite clearly a labour of love and a tremendous engineering feat, involving some really neat tech in the shape of Coccinelle - an OCaml program that accepts "semantic patches" describing program refactorings in a much richer and more general sense than the straightforward line-based diff format.

That said, if I had known three weeks ago what I now know, I would not have spent quite as long trying to get its "integration mode" to work, because it doesn't. What we need to do instead is

• generate a "backport package" using a modern kernel and the scripts in the linux-backports repo. This copies the relevant files from that modern kernel and transforms them into the same code but with lashings of compatibility code and conditionals to make them buildable on any other kernel released since some time in the Neolithic age (3.18? something like that)
• build the modules in that backport package, using as reference the 5.4.64 kernel sources that we got our vmlinux image from

• write something that runs at NixWRT startup to insert those modules in the kernel

The first of these is not super-simple in Nix because gentree.py - the script that does the work - copies a bunch of read-only files from the source tree in /nix/store/ and then tries to overwrite them. I had to do an ugly Python hack to workaround this. Then I did a bunch of other barbarous hacks that may be necesary or may just be because I don't know what I'm doing.

The second is in theory straightforward except for the interaction between CONFIG_FOO settings in the base kernel and CPTCFG_BAR settings in the backported-modules tree that gentree.py has made. In brief, some experimentation is needed to figure out which options need enabling in the base kernel (typically a bunch of crypto code ) to unlock the ability to configure modules in the modules tree that depend on them. I probably still don't have the minimal set, but nor do I have the patience to dig in and find out.

The third is just legwork. I decided to load all the modules at boot time (dynamic device hotplugging is not a supported use case for NixWRT at this time) so I could skip all the work of having NixWRT figure out what the inter-module depenedencies are. Instead I do that myself and load them one at a time in the correct order

I will concede that although modules means more moving parts, this approach does have one big advantage over the monolithic tree it replaces - apart from relieving me of the hassle in trying to e.g. maintain the rt2x00 monster patch - which is that I no longer need the same degree of contortionism to embed device firmware into the kernel. I can just drop it in the filesystem.

A couple of wrong turns were also involved: most notably the day I spent thinking I was building for a GL-MT300N V2 and being puzzled why it wasn't finding the console device, before looking under my desk and realising it was in fact a GL-MT300A (similar but different SoC). Well, I found it funny. Only in retrospect, obviously.

The position right now is that

• I have Mediatek-based wireless extender devices in my study (a.k.a the spare bedroom) and in the "annex" (a.k.a the very large shed at the end of the garden) and they are both working reliably. So far ...

• I have an ath79-based dual radio device (this one ) under my desk (with both radios working!) which is acting as the test device for the PPPoE router/firewall/access point/dns/dhcp server that I have been meaning to get around to since forever and that will finally let me replace the last OpenWrt device around here. I only recently found out that there is a thing called L2TP (which for the purpose of this discussion can be summarised as "PPP over the internet") and that my ISP offers this as a paid service which is awesome because it lets me figure out all the IPv6 and routing malarkey in advance without having to unplug the internet.

Can I speak to your supervisor?#

Fri, 30 Oct 2020 17:22:36 +0000

When we left off last time I was teasing some words abut L2TP ("PPP over the internet") and how it would let me test the actual routing/gatewaying/firewalling etc in NixWRT without breaking the internet access for everyone else in the house.

That's part of what I've been doing since though not exactly all of it. Mostly I have been drowing in alphabet soup.

• L2TP in Linux is implemented with a piece of software called xl2tpd, which creates a udp tunnel between your device (the LAC) and a computer in an ISP somewhere (the LNS). It listens to a unix socket, and once the tunnel is up, you can send some string like c aaisp - supposing you have configured it to know about a LAC called aaisp - to cause it to exchange some L2TP packets on the tunnel and then spawn pppd.

• once pppd is running and has done its own negotiation (PAP! CHAP! BAM! WHIZZ! POW!) then it creates a point-to-point network interface traditionally called ppp0 that allows you to ping the machine at the other end

• now you can run a DHCP and/or DHCP6 client on that interface to get other important details like the routes it provides, networks it has delegated to you, DNS servers, NTP (maybe) and that kind of stuff. OpenWrt have a daemon called odhcp6c which deals with most of this: it runs for the lifetime of the session, and when it gets responses from the servers it is talking to, it calls a script (which you provide) passing the data in environment variables, so that you can run the appropriate commands to confgure your local device.

• the said configuration likely involves yet other services (ntpd, or the dnsmasq you probably run as a dns proxy, or maybe a wireguard vpn, or ...) which need to be configured or reconfigured and started and/or restarted.

Clearly, there are a number of dependencies here (and not forgetting you can't even start the process until you have a transport interface configured for the l2tp to be tunneled over), and modeling these dependencies in Monit was becoming a bit unwieldy. I'm not 100% sure that writing a whole new service supervision system in Lua will be net better, but that's what I've set out to do and early signs are promising. Two sentence summary:

• each service is monitored by a Lua script which starts it and does service-specific health checks on it

• a service publishes its state by writing files into a subdirectory of /run/swarm/services wich it owns, and subscribes to updates from other services using inotify watches on the directories owned by those other services.

It's very much WIP. If I can arrive at a general design that will let me cleanly address (1) the case above; (2) l2tp over lte modem as backup for pppoe; (3) automatically recovering from a buggy ethernet driver/device that panics under heavy load, I will tentatively conclude that it is a good piece of software.

In other news, I have revived (more accurately, rewritten) the QEMU target for NixWRT so now you (and I) can build and run it without access to real hardware. I am only mildly infuriated that QEMU doesn't appear to support device trees on MIPS (or perhaps it does but the Malta MIPS subarch init code doesn't know how to load the dtb that QEMU passes) but it hasn't added a whole lot of complexity to the build to have one target without. See the instructions in the README

If I get much more fed up of having to build a new image and reboot qemu every time I move the bugs around in my Lua scripts, I may also look into adding 9p support so that the emulator can read its root fs direct from the host.