diary @ telent

Stealing file redir from the gods#

Sun Sep 4 12:56:10 2022

Topics: prometheus nix

I'd like to know if my backups are running, so I set up Prometheus and Grafana. There are probably lower-overhead ways to do this, but there are a bunch of other things I think I would also like to track in future.

I'm using restic for backups (NixOS makes this easy) and would like a nice way to run some restic statistics commands, convert the output to a format that Prometheus will parse, and put it somewhere useful.

The Prometheus Way is that it expects to periodically scrape multiple metrics-providing HTTP services and log the outputs in its database. "Proper" restic exporter(s) that work this way do exist - for example, https://github.com/pinpox/restic-exporter - but in order to provide restic statistics it needs to be able to read the restic backup repository, which means either we have a network service running as root or I have to loosen the repository permissions. Also, restic stats takes half an aeon to run: if the stats only change when a backup has taken place, there seems little point in executing it repeatedly whenever Prometheus asks.

So I decided to go with a slightly brute-forcier approach and just run a command at the end of the backup that would dump an appropriately formatted file somewhere the "textfile" collector could read it. Graham Christensen describes how to do this for the system version, and I was able to take a quite similar approach, albeit that there was a bit more parsing required.

I spent most of a morning trying to do this with an unholy combination of mustache and jq and found (1) that I needed to do more transformation than mustache was suited for, and (2) jq couldn't parse dates with millisecond precision. So I cut my losses and wheeled out Ruby - or more specifically, ERB:

<%
  RESTIC = "restic -p #{ENV.fetch("RESTIC_PASSWORD")} -r#{ENV.fetch("RESTIC_REPO")}"
stats = IO.popen(
  "#{RESTIC} stats --json",
  "r") {|f| JSON.parse(f.read) }
snapshots = IO.popen("#{RESTIC} snapshots --json", "r") {|f| JSON.parse(f.read) }
%>
# TYPE total_size gauge
total_size <%= stats["total_size"] %>
# TYPE total_file_count gauge
total_file_count <%= stats["total_file_count"] %>
# TYPE last_snapshot counter
last_snapshot <%= Date.parse(snapshots[-1]["time"]).to_time.to_f %>

We can invoke this template using something like the following incantation -

ruby -r json -r erb -r date -e 'ERB.new($<.read).run' template.erb

so to tie it into the rest of the system and make it run when the backup completes, I call it from the restic backupCleanupCommand.

TIL writeShellApplication which is like writeShellScriptBin but sets up the PATH and has a few other niceties.

# modules/backup/default.nix
{ config, pkgs, ... }:
let
  inherit (pkgs) writeShellApplication;
  passwordFile = "/etc/nixos/secrets/restic-password";
  repository = "/var/spool/backup";
  template = ./template.erb ;
  prom_export = writeShellApplication {
    name = "prom_export";
    runtimeInputs = with pkgs; [ restic ruby ];
    text =''
       mkdir -p /var/lib/prometheus-node-exporter-text-files
       chmod 0775 /var/lib/prometheus-node-exporter-text-files
       RESTIC_PASSWORD=${passwordFile} \
       RESTIC_REPO=${repository} \
       ruby -r json -r erb -r date -e 'ERB.new(File.read(ARGV[0])).run' ${template} \
         > /var/lib/prometheus-node-exporter-text-files/restic.prom.next
       mv  /var/lib/prometheus-node-exporter-text-files/restic.prom.next \
         /var/lib/prometheus-node-exporter-text-files/restic.prom
    '';
  };
in {
  config.services.restic.backups = {
    local =  {
      # your restic config here
      backupCleanupCommand = "${prom_export}/bin/prom_export";
    };
  };
  # etc etc
}

A Short Message from our Server#

Sat Sep 17 12:50:04 2022

Topics: fennel lua ruby nix

Still on the monitoring-things kick, I decided it would be useful to set up alerting. As this network is on the wrong end of a domestic internet connection, I want the alerts to continue to operate even when the internet is unreachable (that might even be one of the things I want to be alerted about!) so I reached for a "wireless broadband" USB dongle with the intent of making it send SMSes.

The Grafana Alerting system out of the box supports a bunch of proprietary systems (Teams, Discord, Hangouts, Slack, Telegram, etc) plus Email, and also "Webhook" - which amounts to "give us a URL and we'll POST a payload to it when things break".

TL;DR I needed to create a web service that sends text messages when people hit it.

Here it is: grafana-sms-alert

I chose to write this in Fennel to see how it holds up to the kind of half-assed sysadminny scripty programming for which I'd usually default to Ruby (and once upon a time, Perl) for. Pretty well, is the answer.

HTTP

Heavy lifting in the HTTP end is performed by lua-http which is way overfeatured for this simple task (even has TLS!) but it's packaged in Nixpkgs so it was an easy choice. The "getting started" section of the docs has an example of using it as a client but not as a server, but happily there's a whole directory of examples in the source distribution that cover that.

JSON parsing is via dkjson which again was chosen mostly just because Nix has already packaged it.

SMS

SMS sending using AT commands is a bit more involved than I expected based on reading a bunch of half-assed examples on the internet - look for AT+CMGS using your favourite web search engine. The problem is that if your message contains any of a number of popular punctuation characters including {, }, [, ], they come out as accented characters. This is because the GSM 7 bit character set only resembles ASCII in some areas, and these symbols aren't in those areas. To encode { (ASCII 0x7b, LEFT CURLY BRACKET) in an SMS message you have to send the bytes 0x1b 0x28, which poses a problem because 0x1b is ESC, which causes the sending to abort.

The fix is to use "SMS PDU Mode" instead of Text Mode. This allows us to encode the message - and the destination phone number, and optionally the SMS service centre - as a long string of hex digits which means we can send any character in the GSM character set including the "extended" ones with the 0x1B prefix. It also requires us to pack the 7 bit characters into octets as a continuous stream (we don't send the high bit, meaning that 160 characters fit into 140 bytes) which required an exciting foray into Lua's bitwise operators. I'm sure this could be done more elegantly and/or more efficiently but I haven't yet figured out how.

Packaging

We use the Nixpkgs Lua Infrastructure to declare the dependencies (luaposix, dkjson, lua-http), then override the Nixpkgs fennel derivation to make it use our lua-with-packages.

Inside a nix-shell we can run fennel main.fnl config.json. But outside of that context we'll need an executable of some form: we use makeWrapper to create a script that runs fennel with the additional flags that tell it where to find the fennel modules that comprise the app.

Starting it as a service is then fairly straightforward: create a module that looks something like the example and update the configuration attributes for your own device path, SMS service centre and phone number.

A note about Huawei devices

Huawei USB stick modems can be configured to operate in a number of modes, and the probability is that when you plug it in you'll get the wrong one - it can manifest as a USB storage device with Windows drivers, or as a network device, or (what we need) as a modem to which we can send AT commands.

After reading far too many web sites on the subject I'm slightly fuzzy about what the "normal" behaviour is, so don't take this as gospel, but what I think happens is

  1. you insert the device, and it is recognised as USB storage - which is probably quite helpful if you're running Windows and otherwise not so much.

  2. if you have usb-modeswitch installed with its default rules, udev notices the new hardware and switches it to network mode.

  3. you try to switch it to TTY mode by running usb_modeswitch manually, which might work if step 2 didn't happen, but otherwise fails because it's already been switched once.

The workaround is to disable usb-modeswitch from running automatically (I leave you to find out how to do this yourself) and then to run it by hand. For me this is

usb_modeswitch -v 0x12d1 -p 0x1f01 -V 0x12d1 -P 0x14dc \
  -M "55534243000000000000000000000011060000000000000000000000000000"

which works on the E3131 (3G dongle) and according to my notes also on the E3372 (LTE). But if it doesn't - well, as I think I already implied, at that point I was in a place of swearing a lot and trying things randomly, so this advice is worth what you paid for it.

⟪ Aug 2022 Oct 2022 ⟫