Secret dogfood#

Sun 13 Sep 2026 06:11:23 PM UTC

Having admitted the other day that I could have written about

rigging up the Liminix wireless extender in the shed to use SOPS for its secrets (unwritten because I ... well, mostly because I just haven't written it)

but hadn't, I felt compelled to sort it out.

I've been working on this on and off for longer than I can remember, because the whole thing still feels brittle and hard to test, and is for that reason unsatisfying. But the other device providing wifi in the shed seems to have entirely ceased working, and it's where I perform $dayjob when I'm not in the office, so I needed to do something as it's embarrassing to have my Zoom calls drop out.

(Every other important device in the shed has wired ethernet, but my work Mac is too locked down and corporate to make that an option)

There are a couple of things I want the new generation of Liminix devices to do that the old ones don't:

• send their logs somewhere that won't disappear on reboot. I have talked about this previously, concluding on that occasion that it was "basically finished and ... just needs installing on some real devices"

• get their "secrets" from a server somewhere instead of baking them into the image at build time, so that secrets can be rotated and configuration changed without reflashing the device. "Secrets" here is currently mostly meaning things like the wifi password, allowed ssh keys etc, which are technically either not secret at all or not especially sensitive but let's at least try hey?

As the first of them was "basically finished", I have been spending hours/months (actual/elapsed) on the second.

In essence it's simple: Liminix already has code to get secrets from a server, which is basically anything that can serve a JSON file over HTTP(S). A service on the client gets the JSON file at configurable intervals, and unpacks it into a directory under /run/services/outputs/service-name where it can be referenced by any other service that needs it

In essence it would be simple, so of course I have accidentally it more complicated, by deciding to keep secrets in SOPS and to use HTTPS client certificates for authentication.

• I wrote an HTTPS server (it's called Undercroft) that does mTLS (could have just used nginx for this bit) and on each request runs sops decrypt (probably couldn't have just used nginx for this bit) to decrypt my secrets file on the fly and transform it from yaml - which is human-readable but not easily machine parseable - to json, which is the opposite.

• On the client side, the Lminix secrets fetcher was previously only capable of HTTP basic auth, but luckily it was dead simple to add mTLS because fetch-freebsd already consults environment variables SSL_CLIENT_CERT_FILE, SSL_CLIENT_KEY_FILE, SSL_CA_CERT_FILE

The threat modelling here might be a little weird, to be honest. using SOPS means that our secrets are encrypted at rest and we can back them up, put them in git servers etc with impunity, etc but instead of having a decryption key per device we decrypt using a single key owned by the https server, and then re-encrypt (courtesy of TLS) for the wire. The more sympathetic way to use SOPS would probably be to use ssh host keys as age identities, but that requires getting the host keys off each of the devices after they've been created at first boot, then re-encrypting the file for all the recipients.

How the client gets its certificate in the first place is the part that's still a bit unsatisfying, because it's automated and the authentication uses a shared secret that's baked into the Liminix image - so anyone with a copy of one of your devices' Liminix images and binwalk could make themselves a TLS cert and fire off a request to your Certifix service to sign it. But when the machine seeking a signature has no identifying features that can't be faked, and no i/o capabilities, how does it attest that it really is the machine it claims to be, and not an imposter?

• ssh host keys are generated on first boot. We don't want to create them on the build machine and bake them into the image, but this means that to see what key was generated we'd have to connect to the box after it booted, meaning (a) that it can't be depending on any of those secrets to configure networking, (b) how do we know we're connecting to the right place?

• the machines are physical computers that can't be certified by their hypervisor (as in AWS, Azure etc)

• the mac address is easy to guess/fake

• we could write some secret into u-boot env, but only on devices where u-boot has been configured with a working persistent environment, and only if we can get to the u-boot prompt (probably needs serial console)

• if the device has controllable LEDs, the server could send it a pattern and it could blink the LEDs in that pattern. I've only just thought of this idea and it might be a good one.

• we could change how images are written and flashed so that a per-device secret that authenticates the machine to certifix is appended/prepended/merged into the image before it's flashed. This might be possible but will require some thought if we want to keep the secret out of the nix store. Ideally we'd make the secret some kind of time-limited, single-use token.

Anyway, I've done none of those things, even the ones that are possible. But it's up and running and I flashed it using Levitate using a chunk of code like this in its configuration

  defaultProfile.packages = with pkgs; [
    (levitate.override {
      config = {
        imports = [ "${modulesPath}/ssh" ];
        programs.busybox.applets = [ "flashcp" ];
        services = {
          # dhcp on wired ethernet only instead of trying to bring up
          # all the wifi stuff
          dhcpc4 =  svc.dhcp4c.client.build {
            interface = config.hardware.networkInterfaces.lan;
            dependencies = [ config.services.hostname ];
          };
          # for recovery image we will just embed static keys instead of
          # running the full secrets management doodad
          sshd = svc.ssh.build {
            authorizedKeys.root = import ../users/dan/authorized-keys.nix;
          };
        };
      };
    })
  ];

There is a gotcha here that I need to address: you have to re-import the ssh module inside levitate's config, otherwise although it finds svc.ssh OK, it doesn't set config.programs.busybox.options.FEATURE_FANCY_ECHO and as a result the authorizedKeys file is garbage.

Anyway, it's up and running and doesn't seem to have crashed yet. There is something weird (but, I think, basically cosmetic) going on with the log shipping forwarder which keeps stopping and restarting whenever it does a DHCP renew, but I will write about that another time - hopefully, after I have found out what the hell is happening.

In other news, Liminix is currently not building against nixpkgs unstable because of a cross-compilation execline build failure. Check the Hydra for the last known working version.