A surfiet of services#
Tue 09 Jul 2024 08:36:58 PM UTC
It feels like a while since I wrote one of these, but this might be because [checks post history] it has indeed been a while since I wrote one of these. Sorry about that. It's been a bit of a slog.
What I've been working on for the past ~~several hundred years~~ couple of months is service failover. Specifically (although, hopefully, the mechanisms developed will be useful more generally): if my PPPoE internet connection goes down, I want to use the LTE modem instead.
There are, as always, details - and anyone else who wants to construct a similar configuration will need to consider whether their details match mine.
-
my LTE modem is a USB thing in which you install a SIM card. Once you have sent it an arcane squence of AT commands, it exposes a wwan network device with an RFC1918 10.0.0.0/8 address. Specifics of that last bit might vary according to the SIM provider/mobile network but (at least in the UK, don't know about other countries) chances are there's NAT everywhere and possibly even CGNAT, there won't be IPV6, no incoming connections etc.
-
but happily, Andrews & Arnold, the fixed-line ISP that I am using for testing Liminix (and also the ISP I use generally and whose praises I sing to anyone even vaguely techy who wants internet access in the UK) provide their service over L2TP - if your DSL line goes down for some reason, you can use a different ISP to connect to them over L2TP using the same credentials, and get issued the same IP address(es), same billing, etc.
-
note that A&A will route your incoming packets to the L2TP connection if it's up, so you can't usefully have PPPoE and L2TP running at the same time.
So, the requirement is to run PPOE until it goes down, then start an L2TP LAC ("L2TP Access Concentrator", the L2TP jargon for the end of a connection that calls the other end) - for which we need some kind of internet connection and a route to the LNS (L2TP Network Server, a.k.a the other end) and some DNS service so we can look up the address of the LNS, and maybe a route to the DNS - although actually I cheated there and assumed it would be on the same network as the wwan device. We can start all of those things in parallel with PPPoE, it's only the actual L2TP service that we have to delay until the primary connection dies.
LTE WWAN
Configuring the modem was a great test of the uevent controlled services I described last time, and I'd say we passed maybe 80%?
-
When the USB stick is plugged in, it identifies as a USB mass storage device with usb id 12d1:14fe. We then need to run USB ModeSwitch which sends it some sequence of magic USB commands (undocumented and reverse-engineered from the Windows drivers)
-
... that causes it to detach from the USB bus and re-attach providing a network device and a serial device, with a different USB id (1201:1506 if you're keeping score). When it reappears, then we open the serial port and send it some (mostly-documented) AT commands to configure APN, username, password, etc
-
then the network is "up" and we can run DHCP to get an IP address and the whereabouts of a nameserver and do whatever else
So we need two uevent controlled services: one to run on initial insertion that does the modeswitch, and the second to run when the "switched" device appears that sends it AT commands. To implement the second we had to extend uevent-watch/devout so that it would match on "sysfs attributes" not only on fields in the uevent message, because there isn't a uevent field that contains the name of the serial port - that information is only in sysfs.
Why do I say 80%? Because there's a kludge. After my dogged insistence in the previous post that this is conceptually better than udev because udev "allows arbitrary commands on each event and it doesn’t have symmetry" - it is fated that I find an apparent need for exactly that. Consider the modeswitch service:
- it runs when usb id 12d1:14fe is present
- therefore, it stops running when usb id 12d1:14fe goes away - which is what modeswitch makes it do. Luckily(?) it doesn't do anything on stop - the mode is never switched back again
- just to reiterate that point, it is no longer running when we need to send AT comands
- therefore the service that sends AT commands cannot have the modeswitch service as a dependency
- but the service that sends AT commands requires the modeswitch service to have run to completion before it starts. This is not actually so bad because it won't run anyway until 1201:1506 appears
- however if there isn't a dependency relationship of some kind between modeswitch and at least some other service, it won't be part of the system closure so won't be included in the image
I am still very much in favour of the constraint that device events
can only start/stop services and not just do whatever they like, but I
don't yet have a good way to model this particular one. The bad
way
is to add the modeswitch service to buildInputs of the AT command
service.
up the tree
We also had to write a new command s6-rc-up-tree to bring up a
service and some but perhaps not all of the services it depends on -
and some logic to avoid bringing those services up on boot. Why?
Suppose you have a network device that's hotplugged and a route
service depending on it. When the device disappears we stop the
service and s6-rc knows how to stop all the services that depend on
it - but when we start the service, s6-rc will start only that
service and not the other services it enables. So we have to go
through all the things that hang off it and start each of them too -
except for the ones that also depend on some other controlled service
if that other controlled service is down.
If that sounds hard to understand, it was hard to understand.
Failover
To accomplish failover we made a new kind of service controller, that
accepts a set of services and runs each of them in turn until it falls
over, then stops it and starts the next one. To detect that a service
has died we watch its outputs directory using inotify to see when it
goes away. We had to make sure the services would actually die when
connectivity is lost:
-
For PPPoE we configure it to send lcp-echo requests and terminate if more than (some number) go unanswered.
-
For L2TP we can configure the daemon to "autodial" an LNS and start a session, and that session is running PPP so we can use lcp-echo as we did with PPPoE, but when the PPP dies it doesn't take the l2tp process with it. We had to patch xl2tpd to terminate when all the endpoints bcome inactive. (We resisted the temptation to patch lxl2tpd in a number of other ways, starting with teaching it to count retries correctly)
One point to watch out for: in s6-rc there is a distinction between a
process being started (fork and exec have happened) and a service
being ready, which the process itself signals to the supervisor by
writing to a particular file descriptor. The s6-rc -u change
command that starts a service will not return until either the service
is ready or a timeout has expired, and until the timeout has elapsed
it will attempt to restart the process - perhaps several times - if it
dies without first becoming ready. That can be confusing if you're
looking at the logs wondering why you see service A, A, A, A being
invoked and failing repeatedly when you were expecting A, B, A, B
... but it's not a bad thing when you know it's happening.
Note there is no automated recovery. It would be simple enough to add an alarm of some kind that kills the secondary service after a decent interval so that the primary is tried again, but that falls squarely in "policy not mechanism" and I haven't decided what policy I want.
Current status: it works on the testbed but I've not integrated it into a full system config yet, and there is oodles of cleanup to do.
In other news
I ran deadnix and nixfmt across the entire Liminix codebase, so now the commas are at the end of the parameters in function argument lists instead of at the start. I didn't merge all the changes that nixfmt wanted to make, just the ones I thought were unambiguous improvements.
Sorry if you had long-running branches that no longer apply.
F is for Folly - specifically, Folly Arch: "a prominent North Mymms landmark since the first half of the 18th century."