diary at Telent Netowrks

Crossing the threshold - Liminix#

Wed, 19 Oct 2022 21:20:32 +0000

I am restarting/rewriting NixWRT, which has seen no real development in, erm, about four years (my, how the time has flown) and is showing its age and showing my Nix inexperience.

līmen (genitive līminis) (neut.)

  1. threshold, doorstep, sill (bottom-most part of a doorway)
    1. lintel
    2. threshold, entrance, doorway, approach; door
    3. house, home, abode, dwelling
    4. beginning, commencement
    5. end, termination

Thus: Liminix, which stands at the threshold of your home network. According to the commit history I've been playing around with it for about a month now (so, since shortly after I broke the family internet for most of a morning while trying to upgrade OpenWrt ), so although it still doesn't actually do anything useful yet perhaps it's time to break cover.

The objectives are quite similar to the NixWRT objectives in that I want to have congruent configuration management on the "infrastructure" devices that make up my home network, and those devices are typically underpowered for running full-blown NixOS. I do though have a shopping list of things I want to do better/differently:

So far: we're using s6-rc for services, which seems to be quite nice and well-put together but I haven't tried too hard to hurt yet. We're using the NixOS module system infra for declaring configuration option types and merging logic. We have significantly more in the way of automated testing than NixWRT had - admittedly not a high bar - and an entirely unrealised/untested idea of how we might do secrets. And the "we" there is, yes, editorial

We don't yet have: writable filesystem (ubifs?); anything o11y; more than one hardware device. And it's not yet at the point that I can dogfood it. Although technically it boots and runs on my spare GL-AR750, I haven't ported wifi across yet.

The primary repo is at https://gti.telent.net/dan/liminix because the older I get the more stubborn I become about free "if you're not paying for it you're the the product" services, but there's a mirror on Github for everyone who's not me. Because federated Gitea is not yet an available thing, and I don't want to throw up all the barriers to contribution.

A Short Message from our Server#

Sat, 17 Sep 2022 13:50:04 +0000

Still on the monitoring-things kick, I decided it would be useful to set up alerting. As this network is on the wrong end of a domestic internet connection, I want the alerts to continue to operate even when the internet is unreachable (that might even be one of the things I want to be alerted about!) so I reached for a "wireless broadband" USB dongle with the intent of making it send SMSes.

The Grafana Alerting system out of the box supports a bunch of proprietary systems (Teams, Discord, Hangouts, Slack, Telegram, etc) plus Email, and also "Webhook" - which amounts to "give us a URL and we'll POST a payload to it when things break".

TL;DR I needed to create a web service that sends text messages when people hit it.

Here it is: grafana-sms-alert

I chose to write this in Fennel to see how it holds up to the kind of half-assed sysadminny scripty programming for which I'd usually default to Ruby (and once upon a time, Perl) for. Pretty well, is the answer.

HTTP

Heavy lifting in the HTTP end is performed by lua-http which is way overfeatured for this simple task (even has TLS!) but it's packaged in Nixpkgs so it was an easy choice. The "getting started" section of the docs has an example of using it as a client but not as a server, but happily there's a whole directory of examples in the source distribution that cover that.

JSON parsing is via dkjson which again was chosen mostly just because Nix has already packaged it.

SMS

SMS sending using AT commands is a bit more involved than I expected based on reading a bunch of half-assed examples on the internet - look for AT+CMGS using your favourite web search engine. The problem is that if your message contains any of a number of popular punctuation characters including {, }, [, ], they come out as accented characters. This is because the GSM 7 bit character set only resembles ASCII in some areas, and these symbols aren't in those areas. To encode { (ASCII 0x7b, LEFT CURLY BRACKET) in an SMS message you have to send the bytes 0x1b 0x28, which poses a problem because 0x1b is ESC, which causes the sending to abort.

The fix is to use "SMS PDU Mode" instead of Text Mode. This allows us to encode the message - and the destination phone number, and optionally the SMS service centre - as a long string of hex digits which means we can send any character in the GSM character set including the "extended" ones with the 0x1B prefix. It also requires us to pack the 7 bit characters into octets as a continuous stream (we don't send the high bit, meaning that 160 characters fit into 140 bytes) which required an exciting foray into Lua's bitwise operators. I'm sure this could be done more elegantly and/or more efficiently but I haven't yet figured out how.

Packaging

We use the Nixpkgs Lua Infrastructure to declare the dependencies (luaposix, dkjson, lua-http), then override the Nixpkgs fennel derivation to make it use our lua-with-packages.

Inside a nix-shell we can run fennel main.fnl config.json. But outside of that context we'll need an executable of some form: we use makeWrapper to create a script that runs fennel with the additional flags that tell it where to find the fennel modules that comprise the app.

Starting it as a service is then fairly straightforward: create a module that looks something like the example and update the configuration attributes for your own device path, SMS service centre and phone number.

A note about Huawei devices

Huawei USB stick modems can be configured to operate in a number of modes, and the probability is that when you plug it in you'll get the wrong one - it can manifest as a USB storage device with Windows drivers, or as a network device, or (what we need) as a modem to which we can send AT commands.

After reading far too many web sites on the subject I'm slightly fuzzy about what the "normal" behaviour is, so don't take this as gospel, but what I think happens is

  1. you insert the device, and it is recognised as USB storage - which is probably quite helpful if you're running Windows and otherwise not so much.
  2. if you have usb-modeswitch installed with its default rules, udev notices the new hardware and switches it to network mode.
  3. you try to switch it to TTY mode by running usb_modeswitch manually, which might work if step 2 didn't happen, but otherwise fails because it's already been switched once.

The workaround is to disable usb-modeswitch from running automatically (I leave you to find out how to do this yourself) and then to run it by hand. For me this is

usb_modeswitch -v 0x12d1 -p 0x1f01 -V 0x12d1 -P 0x14dc \
  -M "55534243000000000000000000000011060000000000000000000000000000"

which works on the E3131 (3G dongle) and according to my notes also on the E3372 (LTE). But if it doesn't - well, as I think I already implied, at that point I was in a place of swearing a lot and trying things randomly, so this advice is worth what you paid for it.

Stealing file redir from the gods#

Sun, 04 Sep 2022 13:56:10 +0000

I'd like to know if my backups are running, so I set up Prometheus and Grafana. There are probably lower-overhead ways to do this, but there are a bunch of other things I think I would also like to track in future.

I'm using restic for backups (NixOS makes this easy) and would like a nice way to run some restic statistics commands, convert the output to a format that Prometheus will parse, and put it somewhere useful.

The Prometheus Way is that it expects to periodically scrape multiple metrics-providing HTTP services and log the outputs in its database. "Proper" restic exporter(s) that work this way do exist - for example, https://github.com/pinpox/restic-exporter - but in order to provide restic statistics it needs to be able to read the restic backup repository, which means either we have a network service running as root or I have to loosen the repository permissions. Also, restic stats takes half an aeon to run: if the stats only change when a backup has taken place, there seems little point in executing it repeatedly whenever Prometheus asks.

So I decided to go with a slightly brute-forcier approach and just run a command at the end of the backup that would dump an appropriately formatted file somewhere the "textfile" collector could read it. Graham Christensen describes how to do this for the system version, and I was able to take a quite similar approach, albeit that there was a bit more parsing required.

I spent most of a morning trying to do this with an unholy combination of mustache and jq and found (1) that I needed to do more transformation than mustache was suited for, and (2) jq couldn't parse dates with millisecond precision. So I cut my losses and wheeled out Ruby - or more specifically, ERB:

<%
  RESTIC = "restic -p #{ENV.fetch("RESTIC_PASSWORD")} -r#{ENV.fetch("RESTIC_REPO")}"
stats = IO.popen(
  "#{RESTIC} stats --json",
  "r") {|f| JSON.parse(f.read) }
snapshots = IO.popen("#{RESTIC} snapshots --json", "r") {|f| JSON.parse(f.read) }
%>
# TYPE total_size gauge
total_size <%= stats["total_size"] %>
# TYPE total_file_count gauge
total_file_count <%= stats["total_file_count"] %>
# TYPE last_snapshot counter
last_snapshot <%= Date.parse(snapshots[-1]["time"]).to_time.to_f %>

We can invoke this template using something like the following incantation -

ruby -r json -r erb -r date -e 'ERB.new($<.read).run' template.erb

so to tie it into the rest of the system and make it run when the backup completes, I call it from the restic backupCleanupCommand.

TIL writeShellApplication which is like writeShellScriptBin but sets up the PATH and has a few other niceties.

# modules/backup/default.nix
{ config, pkgs, ... }:
let
  inherit (pkgs) writeShellApplication;
  passwordFile = "/etc/nixos/secrets/restic-password";
  repository = "/var/spool/backup";
  template = ./template.erb ;
  prom_export = writeShellApplication {
    name = "prom_export";
    runtimeInputs = with pkgs; [ restic ruby ];
    text =''
       mkdir -p /var/lib/prometheus-node-exporter-text-files
       chmod 0775 /var/lib/prometheus-node-exporter-text-files
       RESTIC_PASSWORD=${passwordFile} \
       RESTIC_REPO=${repository} \
       ruby -r json -r erb -r date -e 'ERB.new(File.read(ARGV[0])).run' ${template} \
         > /var/lib/prometheus-node-exporter-text-files/restic.prom.next
       mv  /var/lib/prometheus-node-exporter-text-files/restic.prom.next \
         /var/lib/prometheus-node-exporter-text-files/restic.prom
    '';
  };
in {
  config.services.restic.backups = {
    local =  {
      # your restic config here
      backupCleanupCommand = "${prom_export}/bin/prom_export";
    };
  };
  # etc etc
}

NixOS copy closure#

Sat, 20 Aug 2022 22:09:10 +0000

Breaking blog silence with a short entry, because maybe short entries are easier to write than the long ones I have signally failed to write at any time so far in 2022.

Scenario: you have a computer running NixOS which is too small or too slow for you to really enjoy running nixos-rebuild on, especially if you think it might start trying to build things instead of just copying prebuilt binaries from elsewhere.

(I do not know why, but none of my computers ever seem able to install notmuch without building from source. But that's not important right now)

How can you build its configuration.nix on another machine and then copy everything across? You do something like this:

$ nix-build -E "(import <nixpkgs> {}); (pkgs.nixos (import ./configuration.nix)).config.system.build.toplevel"
$ nix-copy-closure --to root@${TARGET} --include-outputs ./result \
 && ssh root@${TARGET} `readlink result`/bin/switch-to-configuration switch

What's it doing?

  1. build the toplevel derivation (the same one as is used by nixos-rebuild
  2. copy it and anything else it needs to the target system using the magic of nix-copy-closure
  3. run switch-to-configuration on the target system, which does the imperative setup - making files in /etc and /boot and all that stuff - that nixos-rebuild does when you ask it to switch

But Daniel, why is this interesting to you? Do you have such small computers? Yes, reader, I have two mobile phones[*] and one of them - albeit not yet one of them that I'm actually using for telephony or communication purposes - is almost running NixOS

[*] technically 3 but one of them doesn't recognise its SIM any more after I dropped it in the toilet. Until recently I had a Pinephone too, but we parted ways citing creative differences

A Kind of Magic#

Sun, 19 Dec 2021 11:52:12 +0000

Lately my greatest programming success is a Lua program that flashes the backlight on my PineTime smartwatch - or, more accurately, creating the conditions that make it possible to flash the backlight on my smartwatch using Lua. By which I mean, installing Lua on it and writing some bindings for GPIO control.

But this post isn't about PineTime per se, this is about setting up a development environment for it: specifically, about installing Black Magic Probe on the WeAct STM32F411 board commonly known as "Black Pill"

Black Magic Probe is an "in-application debugging tool for embedded microprocessors". You connect one end of it to a USB port on a PC, and the other end to some debug pins (SWDIO, SWCLCK, V+ and GND) on the target device. It appears on the PC as two serial ports, and one of them speaks the GDB remote protocol so you can run programs, set breakpoints, single-step and all that cool stuff on your target device. You can buy the BMP as a hardware device (which I would definitely recommend doing to support the developers, except that it's currently sold out) or you can buy a cheap microcontroller and build and install it yourself.

BMP supports a bunch of different microcontrollers. I read on the Internet(sic) that I should use a "Blue Pill", and I would like to summarise the learning experience that ensued:

  1. Blue Pill is the nickname of a small blue PCB based on the STM32F103 Arm Cortex M3 MCU designed by WeAct Studios, and a zillion clones and variants based on it. You can program it using the Arduino IDE, if that's your idea of a good time, or you can use a "grown up" Arm toolchain. As far as I can tell from reading, to flash it for the first time you need to attach a programmer like the ST-Link (I'm guessing you may also be able to use a Pi with OpenOCD) to its debug pins, but if you flash a program with USB support you can do subsequent uploads over USB.
  2. Some of the Blue Pill clone devices are printed on black PCBs, and therefore known as Black Pill. Same MCU, slightly different board layout.
  3. There is also another device (or range of devices) informally known as the Black Pill, but these use the STM32F401 or STM32F411 - that 4 is significantly not a 1 there. The difference is it's an ARM Cortex M4 rather than M3. There are other changes as well: the USB connector is C rather than micro, they can be USB flashed out of the box, and there are actual buttons instead of jumpers to reset and to enter the bootloader.

The lesson here is: if you want a blue pill, don't assume as I did that something described as "Black Pill [...] better than Blue" is going to do the same job in the same way. Check the MCU model number.

BMP can be built for the F4 MCU, but the documentation isn't very clear on how. Instead of swlink you use the f4discovery target, and you have to pass BLACKPILL=1 to make. And that is alleged to work, but on my Black Pill clone board it didn't until I made a bunch of random changes that shouldn't have made a difference, and then suddenly it did - gory details are in the link. But at least programming the board is simpler: you can do it over USB with dfu-util instead of messing around with serial pins.

For extra credit, and this is a great reason to run BMP instead of using an ST-Link device with OpenOCD, you can enable RTT which gives the target a way to print messages on the host so you can do "printf debugging". At the time of writing this doesn't exist in BMP mainline but there's a PR (see #954) which "just works" and I strongly endorse.

In the next installment, maybe some actual PineTime programming.