Secret Service#

Mon 26 Aug 2024 09:05:39 AM UTC

What I have lately been working on in Liminix is adding the ability to have it use an external source for secrets instead of baking them into the image.

Up until now, the secrets needed by various services (such as wifi passphrases, PPP username/password, ssh keys, etc) are provided as literal values in the configuration.nix (or some other file imported by it) and woven into config files/scripts at buil time. This has consequences:

if you want to put your config in a public git repo, you need to make sure that your secrets are not stored in git. A typical pattern in the examples is to import secrets from a separate file that's named in .gitigore.
all the config files and scripts with embedded secrets are generated using Nix derivations, so they are visible as world-readable files in /nix/store. Whether this matters depends on your threat model: probably you don't have untrusted users on your router, so anyone legitimately logged into to it is already root and would be able to see that data already. But, this could still be an escalation if an attacker gained non-privileged access.
perhaps most importantly, to change any of those secrets, you need to rebuild the configuration. This is annoying but managable if you have a writable filesystem on the device and can use liminix-rebuild, but if it means reflashing then that's more of a faff. Especially if you have a whole bunch of devices across your office and they all need updating with a new SSH key at the same time.

As of now

Currently we have this: last night I added an SSH public key to a JSON file on my development machine, and five minutes later - without my having to do anything else - it was installed as an authorized key for the root user on my Liminix test device.

So, how does it work?

Services, of course. (always_has_been_meme.gif)

Services in Liminix can have outputs. This is a convention we adopted for "discovery" services which provide data that other dependent services might need. For example, PPP and DHCP services discover information like IP addresses, peer addresses, DNS information etc, Other services might use this information for example to create a route to the peer, or to configure upstreams in a stub DNS resolver. Outputs are stored in an ephemeral filesystem (/run/services/outputs) where they can be read using shell in the startup script of another service, or in Fennel with the anoia.svc module that notifies whenever an output changes.

It turns out that (this might have been anticipated a while ago) this is exactly what we need for retrieving external secrets. We write a service that periodically fetches a JSON file-of-all-the-secrets - such as you could export from SOPS, for example - using HTTPS, and writes all the values therein as outputs. Dependent services refer to those outputs.

Job done? There is a little more to it...

First, the dependent services need to be able to actually use that information. We're not storing the entire hostapd.conf in a JSON file, just the wpa passphrase, so the service that starts hostapd first needs to create a suitable configuration file with the secrets embedded in it. To help with this there's output-template which (rather like ERB in Ruby) reads an input file and evaluates delimited sections of Lua code embedded within it. You can see how we use it in the ppp service to make a ppp-options file before starting pppd.

(Some services were easier to adapt than others. pppd took about ten minutes, but for sshd I wrote/adapted a source code patch for dropbear to make it look for its authorized_keys elsewhere than $HOME/.ssh/authorized_keys - because /home is immutable in Liminix and I thought it was cleaner and more robust than a symlink farm.)

Second, when the secrets change then we need to rewrite the config file and (in some/many cases) restart the process. You can see an example of this in the hostapd service - it wraps the actual hostapd in a subscriber service instance that watches the secret service. The subscriber is implemented with watch-outputs, a Fennel script using anoia.svc to get notified whenever the watched service changes.

What's next?

It's not entirely finished, but there's enough of it to post about. Still to do:

Add http basic auth to the secrets fetch.
Add an option to store the secrets locally in an encrypted file, instead of using http(s) - opening up the possibility of configuring a device by e.g. plugging a thumbdrive into it. The plan here is to use clevis/tang for unattended decyption.
I noticed while writing this post that pppd is not wrapped in a subscriber service. Oops. Need to fix that.
Some of the names of things are awful and may change if I can think of better ones.
Write documentation.

In other news

Liminix is broken under Nixpkgs unstable right now, due to changes in how the Lua packaging works. Yes, I plan to fix that. No, I haven't fixed it yet.

(I generally develop Liminix against the latest stable release of nixpkgs, so I'd always suggest trying that first. There is a CI job that builds against unstable but it's not always the first to get fixed.)