Centralised logging with Liminix and VictoriaLogs#

Tue Oct 21 17:52:38 2025

It's a year since I wrote Log off, in which I described some ongoing-at-the-time work to make Liminix devices log over the network to a centralised log repo. It's also, and this is entirely a coincidence, a year since I made any kind of progress on it: since that time all my log messages have continued to be written to ramdisk that will be lost forever like tears in the rain.

This situation was not ideal. I had some time and energy recently to see if I could finish it up and, well, I haven't done that exactly but whereas last time I only believed it was substantially finished, this time I believe it is substantially finished.

It goes a little something like this:

Tap the log pipeline

Each service in Liminix is connected to its own log process, which is (for 98% of the services) connected to the "fallback logger" which writes the logs to disk (ramdisk) and takes care of log rotation etc. This is standard s6 stuff, we're not innovating here.

Into the middle of this pipeline we insert a program called logtap which copies its input to its output and also to a fifo - but only writes to the fifo if the previous writes worked (i.e. it doesn't back up or stop working if the tap is not connected). The standard output from logtap goes on to the default logger, so local logging is unaffected - which is important if the network is down or hasn't come up yet.

This is a change from last year's version, which used a unix domain socket instead of a fifo. Two reasons: first, we need to know which messages were sent successfully and which weren't. It was difficult to tell reliably and without latency whether there was anything at the other end of the socket, whereas we learn almost instantly when a fifo write fails. Second, it makes it easier to implement a shipper because it can just open the fifo and read from it, instead of having to call socket functions.

Hang a reader on the tap

The log shipper opens the other end of the fifo and ... ships the logs. I've chosen VictoriaLogs (wrapped in an HTTPS reverse proxy) as my centralised log service, so my log shipper has to conect with HTTPS to the service endpoint and send "jsonline" log messages. In fact, my log shipper just speaks pidgin HTTP on file descriptors 6 and 7 and leverages s6-tlsclient to do the actual TCP/TLS heavy lifting.

This is all new since last year when we were just splatting raw logs over a socket connection instead of doing this fancy JSON stuff. It did mean writing a parser for TAI64N external timestamps and some functions to convert it to UTC: as a matter of principle (read: stubbornness) I do appreciate that my log message timestamps won't go forwards and backwards arbitrarily when leap seconds are decreed, but I guess almost nobody else (at least, neither VictoriaLogs nor Zinc) thinks it's important.

  # in liminix config
  logging.shipping = {
    enable = true;
    command =
      let certc = config.services.client-cert;
      in ''
        export CERTFILE=$(output_path ${certc} certificate)
        export CAFILE=$(output_path ${certc} ca-certificate)
        export KEYFILE=$(output_path ${certc} key)
        ${pkgs.s6-networking}/bin/s6-tlsclient -j -y -k loghost.example.org \
          10.0.0.1 443 \
          ${pkgs.logshippers}/bin/victorialogsend https://loghost.example.org/insert/jsonline
      '';
    dependencies = [services.qemu-hyp-route services.client-cert];
  };

... using the TLS cert you previously requested

Before the log shipper can start, it needs to get its TLS client certificate, by making a CSR and sending it to Certifix. The certifix-client is almost the same as last year's version except that it uses lua-http instead of fetch-freebsd as the http interface. This is because last year's version wasn't work when asked to traverse the baroque maze of iptables forwarding and QEMU Slirp networking that lies between my Liminix test network and my VictoriaLogs instance. After a long time staring at pcap dumps I gave up trying to work out why and just rewrote that bit.

It's important to have an (at least vaguely) accurate clock before attempting HTTPS, because the server certificate has a "not valid before" field, so OpenSSL won't like it if you say it's still 1970.

  # in liminix config
  services.client-cert = svc.tls-certificate.certifix-client.build {
    caCertificate = builtins.readFile /var/lib/certifix/certs/ca.crt;
    subject = "C=GB,ST=London,O=Example Org,OU=devices,CN=${config.hostname}";
    secret = builtins.readFile /var/lib/certifix/challengePassword;
    serviceUrl = "https://loaclhost.lan:19613/sign";
    dependencies = [ config.services.ntp ] ;
  };

... to connect to an HTTPS reverse proxy

Originally I planned to put a Lets Encrypt cert in front of Victorialogs, but that would need 500k of CA certificate bundle on each device, which is quite a lot on devices with little flash. So it makes more sense to use the Certifix CA here too.

Persuading the OpenSSL command line tools to make a CSR with a challengePassword was probably as much work as writing something with luaossl would have been - it was certainly messier - but the point is I didn't know that when I started.

  # in nixos configuration.nix
  systemd.services."loghost-certificate" =
    let
      dir = "/var/lib/certifix";
      pw = builtins.readFile "${dir}/private/challengePassword";
    in {
      script = ''
        set -eu
        cd ${dir}
        PATH=${pkgs.openssl}/bin:${pkgs.curl}/bin:$PATH
        openssl req -config <(printf '[req]\nprompt=no\nattributes=attrs\ndistinguished_name=DN\n[DN]"C=GB\nST=London\nO=Example Org\nCN=loghost\n[attrs]\nchallengePassword=${pw}') -newkey rsa:2048  -addext "extendedKeyUsage = serverAuth" -addext "subjectAltName = DNS:loghost.lan,DNS:loghost,DNS:loghost.example.org" -nodes -keyout private/loghost.key --out certs/loghost.csr
        curl --cacert certs/ca.crt -H 'content-type: application/x-pem-file' --data-binary @certs/loghost.csr https://localhost:19613/sign -o certs/loghost.crt
      '';
      serviceConfig = {
        Type = "oneshot";
        User = "root";
        ReadWritePaths = ["/var/lib/certifix"];
        StateDirectory = "certifix";
      };
      startAt = "monthly";
    };

The proxy itself is just Nginx with ssl_verify_client set, but certifix-client holds the https connection open so remember to disable proxy buffering or you aren't getting your logs in any kind of timely fashion.

  # in nixos configuration.nix
  services.nginx.virtualHosts."loghost.example.org" = {
    forceSSL = true;
    sslTrustedCertificate = /var/lib/certifix/certs/ca.crt;
    sslCertificateKey = "/var/lib/certifix/private/loghost.key";
    sslCertificate = "/var/lib/certifix/certs/loghost.crt";

    extraConfig = ''
      ssl_verify_client on;
      proxy_buffering off;
      proxy_request_buffering off;
    '';

    locations."/".proxyPass = "http://127.0.0.1:9428/";
  };

Just as I did last year, I'm going to finish by claiming that this is basically finished and it just needs installing on some real devices. Hopefully I'm right this time, though.

"sanitize" is a code smell#

Sun Sep 7 08:46:27 2025

This is something of a hobby horse of mine, so forgive the rant: when I see something has been "sanitized" I treat it as a code smell (per Martin Fowler, "... a surface indication that usually corresponds to a deeper problem in the system"), and often find it reveals sloppy thinking which may not even prevent the exploits it is supposed to guard against.

Each data item in your system is a value, which has a canonical representation inside your system but may be represented in multiple different external formats at the boundaries of your system.

When we say "sanitize" we imply that the input data was "insanitary" (or even "insane", same etymological root I think) but it really probably wasn't - it just didn't conform to the rules of some particular representation you had in mind that you would later need to output. So why is that particular representation special? Should "sanitizing" strip out backticks (specal in shell)? The semicolon (special in SQL)? The angle brackets (HTML)? The string +++ (Hayes modem commands)? .. (pathnames)? ` The dollar sign (bound to be used somewhere)? Non-ASCII unicode characters (can't put those in a domain name)?

Don't "sanitize". Encode and decode between the canonical internal representation and the external representation you need to interface with. Mr O'Leary will be happy, Sigur Rós will appreciate you've spelled their name right, and Smith & Sons, Artisan Greengrocers won't have their ampersand dropped.

Respinning threads from Gotosocial#

Mon Aug 25 16:46:35 2025

I made a small Fennel script to extract the text from a GoToSocial thread and splat it into a text file. I have used it once and it seemed to work, so I'm going to explain it a bit.

It needs lua-http, rxi's json.lua, and net-url aka net.url aka neturl, all of which you can currently get from MS Github, if they haven't already replaced the whole site with an AI-mediated barrel of slop by the time you read this.

(local request (require :http.request))
(local url (require :net.url))
(local json (require :json))
(local {: view } (require :fennel))

lua-http might be a bit overkill here but I've used before and I went with what I know. I wrapped it here with a little send-request function so I didn't have underscores all over the place.

The last time I really dug into HTTP was about 2003, so this HTTP/2 concept that the request method is actually a header field with a colon in front of its name is ... unusual and new to me.

The url passed into this method comes from net-url and is actually a table with a metatable that defines __tostring. This is like a string when you print it, but not quite enough like a string when you pass it to lua-http, hence the cast.

(fn send-request [url method headers body]
  (let [r (request.new_from_uri (tostring url))
        h r.headers]
    (h:upsert ":method" method)
    (when body
      (r:set_body body))
    (each [k v (pairs headers)]
      (h:append k v))
    (let [(headers stream) (r:go)]
      (stream:get_body_as_string))))

(fn json-request [url method headers attrs]
  (let [body (json.encode attrs)]
    (-> (send-request url method headers body)
        json.decode)))

The next bit is pretty much a straight translation of the Gotosocial API login flow documentation from bash + curl into Fennel.

One surprise here was that the net-url API is very side-effecty. If you write

(let [u (url.parse "HTTPS://example.com")]
  (print (/ u "some" "path" "segments")))

It will print https://example.com/some/path/segments, as you might expect from skimming the docs, but it will also change the value of u, as you might not expect unless you'd actually read the docs (guess who didn't and only skimmed them). u:resolve is a more functional alternative.

(fn oauth-new-client [root-url client-name]
  (json-request
   (.. (root-url:resolve "/api/v1/apps"))
   :POST
   { :content-type "application/json" }
   {
    "client_name" "fetch-thread"
    "redirect_uris" "urn:ietf:wg:oauth:2.0:oob"
    "scopes" "read"
    }))

(fn oauth-access-token [root-url client_id client_secret code]
  (let [body {
	      : client_id
	      : client_secret
	      :redirect_uri "urn:ietf:wg:oauth:2.0:oob"
	      : code
	      :grant_type "authorization_code"
              }]
    (json-request
      (.. (root-url:resolve "/oauth/token"))
      :POST
      { :content-type "application/json" }
      body)))

(fn request-api-token [root-url]
  (let [{: client_id : client_secret } (oauth-new-client root-url "unroll.fnl")
        redirect-url (..
                      root-url
                      "/oauth/authorize?client_id=" client_id
                      "&redirect_uri=urn:ietf:wg:oauth:2.0:oob&response_type=code&scope=read")]
    (io.stdout:write "login to the instance and then paste in the token here\n> ")

    ;; hardcoded firefox because `xdg-open` doesn't work for
    ;; me right now. That's something to fix another day.
    (os.execute (string.format "firefox %q" redirect-url))
    (let [code (io.stdin:read)
          {: access_token}
          (oauth-access-token root-url client_id client_secret code)]
      access_token)))

The net result of all that code will return an access token.

The access token can be used with the other API calls but not, as I originally thought it would, for fetching actual statuses like /users/dan/statuses/01K2Q3MHSGMHB2CFTRCE6FBP5T. That needs HTTP signatures (a.k.a "authorized fetch") which is something quite else and looks more involved than I wanted to get into at 12:30am.

However, we can fetch the status content in a different format using an API endpoint: /api/v1/statuses/01K2Q3MHSGMHB2CFTRCE6FBP5T gets the post itself and ..../context gets its ancestor and descendant statuses.

(fn fetch-post [root-url headers id]
  (let [url (root-url:resolve (.. "/api/v1/statuses/" id))]
    (json-request url :GET headers nil)))

(fn fetch-post-context [root-url headers id]
  (let [url (root-url:resolve (.. "/api/v1/statuses/" id "/context"))]
    (json-request url :GET headers nil)))

Our conversion to markdown is extremely rudimentary:

(fn format-post [post]
  (each [_ img (ipairs post.media_attachments)]
    (print (string.format "![%s](%s)" img.description img.url)))
  (print post.text))

and the rest is just calling things in order.

• To avoid having to interactively login every time we run the script, we write the access token to a file
• my blog software goes wonky if posts don't have titles, so make one up
• the script is called with a status URL, so we extract the id for the status by assuming it's the last part of the url path

(fn extract-title [text]
  (->
   (or (string.match text "(.-)[%.%?%!\n]") text)
   (string.sub 1 60)))

(fn root-and-id [url-str]
  (let [u (url.parse url-str)]
    (values
     (u:resolve "/")
     (string.match u.path ".+/(.-)$")
     )))

(let [dotfile (string.format "%s/.rethread" (os.getenv "HOME"))
      (root-url post-id) (root-and-id (. arg 1))
      f (io.open dotfile :r)
      api-token
      (if f
          (f:read)
          (let [t (request-api-token root-url)]
            (with-open [out (io.open dotfile :w)]
              (out:write t)
              t)))
      headers { :authorization (.. "Bearer " api-token)}
      context  (fetch-post-context root-url headers post-id)
      post (fetch-post root-url headers post-id)
      first-post (or (. context.ancestors 1) post)]
  (print (.. "Title: " (extract-title first-post.text)))
  (print (.. "Slug: " post.id))
  (print (.. "Date: " (. first-post  :created_at)))
  (print "\n")
  (each [_ p (ipairs context.ancestors)]
    (format-post p))
  (format-post post)
  (each [_ p (ipairs context.descendants)]
    (format-post p)))

Aftercare

Some thoughts as they occur to me:

• I said in 2020 that I no longer have a favourite programming language, but maybe five years later I do again and it's Fennel? It would be nice if I could have LuaJIT and Lua 5.4 at the same time but ... there's always something we can't have

• It's quite hacky: there is no error checking. If it doesn't work, add printf debugging.

• It's quite hacky: it has no tests. I usually prefer to code test-first, but this isn't. That's because it's 90% made of API client glue and 10% trivial, which is the kind of scenario where I've never had value from test harness infra that I couldn't get by running it.

• It's quite hacky: we don't check if the access token has expired. If the access token expires, delete ~/.rethread and run the script again.

• Writing this blog entry has provided an opportunity to learn that the bare url markup in Cripslock is buggy, so yay for finding that out, I guess.

• I would use lua-http and rxi/json again, but net-url is not (yet?) the perfect beautiful gem of programming I could wish for. I'd like something a bit closer to (dare I say it) Ruby, for example, where the URL objects are value objects instead of being mutated in place.

All of that is a very long-winded way of saying "expect to see more blog posts that are really just recycled fediverse threads", but I'm now on my fourth fediverse server and I won't put up again with saying goodbye to all my posts every time I change to a new instance. So, I am blogging for persistence (and some day, less shitty search).

Matching Green#

Sun Aug 3 18:58:47 2025

I've ridden this route three times. The first time I started out following a Kurviger round trip route and then I was having so much fun on the B184 that I refused to take the left turn that Kurviger wanted me to take[*], and instead I followed signposts for a while. When I got home I spent some time with a map and an open tab on Google Streetview and figured I had approximately ridden Abridge - Stanford Rivers - Chipping Ongar - Fyfield - some Rodings - Hatfield Heath - Matching Green - ("Watery Lane, narrow road with gravel in the middle) - Moreton - Bovinger - Tyler's Green - North Weald - Epping and then home, and and almost all of it - all the bits that weren't Epping rush rour - I would have gladly ridden again. A lot looked vaguely familiar (although backwards) from riding the Dun Run, albeit it didn't look that similar because daylight.

After the second time around - a bit faster, because I had no wrong turns or need of pulling over to look at the map - I looked at the Hundred Parishes website and found I'd passed through at least one of the place on their list, so I decided to ride the whole thing a third time and inaugurate the 100-parishes topic.

This is the green in Matching Green. It has

• lots of grass
• trees
• a pond: very little water currently, no ducks that I saw (I may have missed them)
• a cricket pitch and pavilion

The green is overlooked by thatched cottages and a pub. There are traditional stripey signposts in the area

Photos are of the cricket pavilion, if that's the right word, and the weathervane on the roof, depicting a man with a scythe and what appears to be (but probably isn't) a walking frame.

My motorcycle is not thematically appropriate to the "green and rural" theme in this thread but I am going to include this photo anyway (1) to show I was there and didn't lift the pictures from the internet; (2) because it is IMO a very pretty motorbike.

[*] https://tueb.telent.net/w/jiq1UZs4mc38mSeuB9N1wS this is what Kurviger wanted to save me from.

Voltage-sensitive relay: I tried it so you don't have to#

Sat Aug 2 14:46:55 2025

In 2001, "auxillary power" was not a concern of the manufacturers of sportsbikes - even otherwise practical sportsbikes like the CBR600F. It doesn't come with any handy USB or 12V sockets or even spare fuseholders under the seat.

In 2025 I've resisted bedecking it with a million current draws, but I do need power for the USB connector that my phone is connected to, and for the wired-in dashcams, and I thought I'd try and be clever.

The idea was a good one. As eny fule kno, a car/bike battery puts out around 12V and an alternator provides 14V, so if you have accessories that you want to work only when the engine's running and you're not at risk of draining the motorbike battery, you could install a voltage-sensitive relay so that they're only powered when the supply voltage exceeds, say, 13V. Therefore:

• an Aliexpress clone of the Oxford accessory fuse box, this has 4 5A fused connections through relays, and turns them on when it sees a voltage on the thin red wire that you're supposed to wire to a switched live

• but instead of finding a relatively exposed accessory 12v supply somewhere in the loom and tapping it, I connected the thin red wire to a 20A voltage-sensitive relay intended for charging caravan batteries only when the car alternator is running.

So, my accessories get power when the supply voltage exceeds 13V, and cut out again when it drops to 12.2. Which seems on the low side, but there's a little pot in the VSR to adjust it.

Basically I'd made a Healtech Thunderbox clone but half the price and four times the ugliness.

After a few rides I concluded that it doesn't actually work very well though, because of the finickiness of the threshold. You see, both those voltages quoted above are nominal. A fully charged battery could be pumping out as much as 12.8V, and - on my bike at least - when the engine is at idle speed and the headlights are on and the radiator fan is running, the alternator voltage is not much more (or possibly even less). So, my satnav device, which is quite an elderly Android phone with not much life left in its own battery, wasn't getting power at low speeds. And whhen i got home after a ride and took the seat cover off to take stuff out of storage, I could see the LEDs merrily glowing away.

So, tl;dr I took it came out again. The fusebox is still there, but I took the voltage-sensitive magic out and now it just connects to a wire spliced into the taillight Result: now my USB cable turns on when the ignition switch and the lights are on[*], not when the voltage rises above a notional 13V.

You might be able to see in the picture there are three inline fuse holders. Yes, I actually have more things that need an always-on power connection (dashcam, optimate, and the accessory fuse box itself) than I have accessories.

Someone should make a secondary fuse box that has both switched and unswitched connections.

[*] my bike is old enough to actually have a switch to turn the headlight off. I pretty much never use it, though, unless it's been standing for a long time and I'm worried it won't start with the added power drain from the headlights.