Self-ghosting email#

Tue, 21 Mar 2023 22:13:55 +0000

[ Reminder: more regular updates on what I'm spending most of my time on lately are at https://www.liminix.org/ ]

I had occasion recently to set up some mailing lists and although the subject matter for those lists is Liminix-relevant, the route to their existence really isn't. So, some notes before I forget:

the NixOS Manual has a section on Mailman which is a good start - though I didn't copy the ssl-relateddirectives because I already have an SSL cert for my mailserver
but it broke my mail delivery. It turns out that this is because my Postfix configuration is a few years old and doesn't set local_recipient_maps - although the default behaviour works OK, setting it explicitly for Mailman overrides that default instead of ignoring it. So we need to add them back:
```
      local_recipient_maps = [
        "proxy:unix:passwd.byname"
        "$alias_maps"
        "hash:/var/lib/mailman/data/postfix_lmtp"
      ];
```
next we consult the upstream Mailman docs
- most of the stuff in Configuring Mailman Core has been set up already, but it's worth reading the section on Configuring MTA to see what it's done.
- next look at Configure Web Frontend - again, most of this is already done except for mailman-web createsuperuser, which has to be done as the mailman-web user.
to get HTTPS working on the mailman web interface, the nginx virtualHost needs enableACME and forceSSL
now we point a web browser at lists.example.org/admin and login as thesuperuser. Per the instructions in the FAQ we had to change (not delete! deleting will break everything! don't ask me how I know that :sad-angry-face:) the domain name and display name to something real.
now point a web browser at lists.example.org/ and we can start adding the actual lists

Anyway, that's where we are. I'm quite certain I've done something wrong, but I'm yet to discover what.

Sub-liminix messaging#

Wed, 15 Feb 2023 22:23:48 +0000

I am restarting/rewriting NixWRT,

he said, a few months ago. This is a short follow-up announcement to say that

I registered a domain name: you can now find regular Liminix updates at www.liminix.org
there will be regular Liminix updates, because Liminix is now funded by NLnet through the NGI0 Entrust Fund

I am very stoked about this. I'm aiming for ~ weekly updates in that place.

Crossing the threshold - Liminix#

Wed, 19 Oct 2022 21:20:32 +0000

I am restarting/rewriting NixWRT, which has seen no real development in, erm, about four years (my, how the time has flown) and is showing its age and showing my Nix inexperience.

līmen (genitive līminis) (neut.)
threshold, doorstep, sill (bottom-most part of a doorway)
lintel
threshold, entrance, doorway, approach; door
house, home, abode, dwelling
beginning, commencement
end, termination

Thus: Liminix, which stands at the threshold of your home network. According to the commit history I've been playing around with it for about a month now (so, since shortly after I broke the family internet for most of a morning while trying to upgrade OpenWrt ), so although it still doesn't actually do anything useful yet perhaps it's time to break cover.

The objectives are quite similar to the NixWRT objectives in that I want to have congruent configuration management on the "infrastructure" devices that make up my home network, and those devices are typically underpowered for running full-blown NixOS. I do though have a shopping list of things I want to do better/differently:

a writable filesystem so that software updates or reconfiguration (e.g. changing passwords) don't require taking the device offline to reflash it.
more flexible service management with dependencies, to allow configurations such as "route through PPPoE if it is healthy, with fallback to LTE"
a spec for valid configuration options (a la NixOS module options) to that we can detect errors at evaluation time instead of producing a bad image.
a network-based mechanism for secrets management so that changes can be pushed from a central location to several Liminix devices at once
send device metrics and logs to a monitoring/alerting/o11y infrastructure

So far: we're using s6-rc for services, which seems to be quite nice and well-put together but I haven't tried too hard to hurt yet. We're using the NixOS module system infra for declaring configuration option types and merging logic. We have significantly more in the way of automated testing than NixWRT had - admittedly not a high bar - and an entirely unrealised/untested idea of how we might do secrets. And the "we" there is, yes, editorial

We don't yet have: writable filesystem (ubifs?); anything o11y; more than one hardware device. And it's not yet at the point that I can dogfood it. Although technically it boots and runs on my spare GL-AR750, I haven't ported wifi across yet.

The primary repo is at https://gti.telent.net/dan/liminix because the older I get the more stubborn I become about free "if you're not paying for it you're the the product" services, but there's a mirror on Github for everyone who's not me. Because federated Gitea is not yet an available thing, and I don't want to throw up all the barriers to contribution.

A Short Message from our Server#

Sat, 17 Sep 2022 13:50:04 +0000

Still on the monitoring-things kick, I decided it would be useful to set up alerting. As this network is on the wrong end of a domestic internet connection, I want the alerts to continue to operate even when the internet is unreachable (that might even be one of the things I want to be alerted about!) so I reached for a "wireless broadband" USB dongle with the intent of making it send SMSes.

The Grafana Alerting system out of the box supports a bunch of proprietary systems (Teams, Discord, Hangouts, Slack, Telegram, etc) plus Email, and also "Webhook" - which amounts to "give us a URL and we'll POST a payload to it when things break".

TL;DR I needed to create a web service that sends text messages when people hit it.

Here it is: grafana-sms-alert

I chose to write this in Fennel to see how it holds up to the kind of half-assed sysadminny scripty programming for which I'd usually default to Ruby (and once upon a time, Perl) for. Pretty well, is the answer.

HTTP

Heavy lifting in the HTTP end is performed by lua-http which is way overfeatured for this simple task (even has TLS!) but it's packaged in Nixpkgs so it was an easy choice. The "getting started" section of the docs has an example of using it as a client but not as a server, but happily there's a whole directory of examples in the source distribution that cover that.

JSON parsing is via dkjson which again was chosen mostly just because Nix has already packaged it.

SMS

SMS sending using AT commands is a bit more involved than I expected based on reading a bunch of half-assed examples on the internet - look for AT+CMGS using your favourite web search engine. The problem is that if your message contains any of a number of popular punctuation characters including {, }, [, ], they come out as accented characters. This is because the GSM 7 bit character set only resembles ASCII in some areas, and these symbols aren't in those areas. To encode { (ASCII 0x7b, LEFT CURLY BRACKET) in an SMS message you have to send the bytes 0x1b 0x28, which poses a problem because 0x1b is ESC, which causes the sending to abort.

The fix is to use "SMS PDU Mode" instead of Text Mode. This allows us to encode the message - and the destination phone number, and optionally the SMS service centre - as a long string of hex digits which means we can send any character in the GSM character set including the "extended" ones with the 0x1B prefix. It also requires us to pack the 7 bit characters into octets as a continuous stream (we don't send the high bit, meaning that 160 characters fit into 140 bytes) which required an exciting foray into Lua's bitwise operators. I'm sure this could be done more elegantly and/or more efficiently but I haven't yet figured out how.

Packaging

We use the Nixpkgs Lua Infrastructure to declare the dependencies (luaposix, dkjson, lua-http), then override the Nixpkgs fennel derivation to make it use our lua-with-packages.

Inside a nix-shell we can run fennel main.fnl config.json. But outside of that context we'll need an executable of some form: we use makeWrapper to create a script that runs fennel with the additional flags that tell it where to find the fennel modules that comprise the app.

Starting it as a service is then fairly straightforward: create a module that looks something like the example and update the configuration attributes for your own device path, SMS service centre and phone number.

A note about Huawei devices

Huawei USB stick modems can be configured to operate in a number of modes, and the probability is that when you plug it in you'll get the wrong one - it can manifest as a USB storage device with Windows drivers, or as a network device, or (what we need) as a modem to which we can send AT commands.

After reading far too many web sites on the subject I'm slightly fuzzy about what the "normal" behaviour is, so don't take this as gospel, but what I think happens is

you insert the device, and it is recognised as USB storage - which is probably quite helpful if you're running Windows and otherwise not so much.
if you have usb-modeswitch installed with its default rules, udev notices the new hardware and switches it to network mode.
you try to switch it to TTY mode by running usb_modeswitch manually, which might work if step 2 didn't happen, but otherwise fails because it's already been switched once.

The workaround is to disable usb-modeswitch from running automatically (I leave you to find out how to do this yourself) and then to run it by hand. For me this is

usb_modeswitch -v 0x12d1 -p 0x1f01 -V 0x12d1 -P 0x14dc \
  -M "55534243000000000000000000000011060000000000000000000000000000"

which works on the E3131 (3G dongle) and according to my notes also on the E3372 (LTE). But if it doesn't - well, as I think I already implied, at that point I was in a place of swearing a lot and trying things randomly, so this advice is worth what you paid for it.

Stealing file redir from the gods#

Sun, 04 Sep 2022 13:56:10 +0000

I'd like to know if my backups are running, so I set up Prometheus and Grafana. There are probably lower-overhead ways to do this, but there are a bunch of other things I think I would also like to track in future.

I'm using restic for backups (NixOS makes this easy) and would like a nice way to run some restic statistics commands, convert the output to a format that Prometheus will parse, and put it somewhere useful.

The Prometheus Way is that it expects to periodically scrape multiple metrics-providing HTTP services and log the outputs in its database. "Proper" restic exporter(s) that work this way do exist - for example, https://github.com/pinpox/restic-exporter - but in order to provide restic statistics it needs to be able to read the restic backup repository, which means either we have a network service running as root or I have to loosen the repository permissions. Also, restic stats takes half an aeon to run: if the stats only change when a backup has taken place, there seems little point in executing it repeatedly whenever Prometheus asks.

So I decided to go with a slightly brute-forcier approach and just run a command at the end of the backup that would dump an appropriately formatted file somewhere the "textfile" collector could read it. Graham Christensen describes how to do this for the system version, and I was able to take a quite similar approach, albeit that there was a bit more parsing required.

I spent most of a morning trying to do this with an unholy combination of mustache and jq and found (1) that I needed to do more transformation than mustache was suited for, and (2) jq couldn't parse dates with millisecond precision. So I cut my losses and wheeled out Ruby - or more specifically, ERB:

<%
  RESTIC = "restic -p #{ENV.fetch("RESTIC_PASSWORD")} -r#{ENV.fetch("RESTIC_REPO")}"
stats = IO.popen(
  "#{RESTIC} stats --json",
  "r") {|f| JSON.parse(f.read) }
snapshots = IO.popen("#{RESTIC} snapshots --json", "r") {|f| JSON.parse(f.read) }
%>
# TYPE total_size gauge
total_size <%= stats["total_size"] %>
# TYPE total_file_count gauge
total_file_count <%= stats["total_file_count"] %>
# TYPE last_snapshot counter
last_snapshot <%= Date.parse(snapshots[-1]["time"]).to_time.to_f %>

We can invoke this template using something like the following incantation -

ruby -r json -r erb -r date -e 'ERB.new($<.read).run' template.erb

so to tie it into the rest of the system and make it run when the backup completes, I call it from the restic backupCleanupCommand.

TIL writeShellApplication which is like writeShellScriptBin but sets up the PATH and has a few other niceties.

# modules/backup/default.nix
{ config, pkgs, ... }:
let
  inherit (pkgs) writeShellApplication;
  passwordFile = "/etc/nixos/secrets/restic-password";
  repository = "/var/spool/backup";
  template = ./template.erb ;
  prom_export = writeShellApplication {
    name = "prom_export";
    runtimeInputs = with pkgs; [ restic ruby ];
    text =''
       mkdir -p /var/lib/prometheus-node-exporter-text-files
       chmod 0775 /var/lib/prometheus-node-exporter-text-files
       RESTIC_PASSWORD=${passwordFile} \
       RESTIC_REPO=${repository} \
       ruby -r json -r erb -r date -e 'ERB.new(File.read(ARGV[0])).run' ${template} \
         > /var/lib/prometheus-node-exporter-text-files/restic.prom.next
       mv  /var/lib/prometheus-node-exporter-text-files/restic.prom.next \
         /var/lib/prometheus-node-exporter-text-files/restic.prom
    '';
  };
in {
  config.services.restic.backups = {
    local =  {
      # your restic config here
      backupCleanupCommand = "${prom_export}/bin/prom_export";
    };
  };
  # etc etc
}