Stealing file redir from the gods#

Sun, 04 Sep 2022 13:56:10 +0000

I'd like to know if my backups are running, so I set up Prometheus and Grafana. There are probably lower-overhead ways to do this, but there are a bunch of other things I think I would also like to track in future.

I'm using restic for backups (NixOS makes this easy) and would like a nice way to run some restic statistics commands, convert the output to a format that Prometheus will parse, and put it somewhere useful.

The Prometheus Way is that it expects to periodically scrape multiple metrics-providing HTTP services and log the outputs in its database. "Proper" restic exporter(s) that work this way do exist - for example, https://github.com/pinpox/restic-exporter - but in order to provide restic statistics it needs to be able to read the restic backup repository, which means either we have a network service running as root or I have to loosen the repository permissions. Also, restic stats takes half an aeon to run: if the stats only change when a backup has taken place, there seems little point in executing it repeatedly whenever Prometheus asks.

So I decided to go with a slightly brute-forcier approach and just run a command at the end of the backup that would dump an appropriately formatted file somewhere the "textfile" collector could read it. Graham Christensen describes how to do this for the system version, and I was able to take a quite similar approach, albeit that there was a bit more parsing required.

I spent most of a morning trying to do this with an unholy combination of mustache and jq and found (1) that I needed to do more transformation than mustache was suited for, and (2) jq couldn't parse dates with millisecond precision. So I cut my losses and wheeled out Ruby - or more specifically, ERB:

<%
  RESTIC = "restic -p #{ENV.fetch("RESTIC_PASSWORD")} -r#{ENV.fetch("RESTIC_REPO")}"
stats = IO.popen(
  "#{RESTIC} stats --json",
  "r") {|f| JSON.parse(f.read) }
snapshots = IO.popen("#{RESTIC} snapshots --json", "r") {|f| JSON.parse(f.read) }
%>
# TYPE total_size gauge
total_size <%= stats["total_size"] %>
# TYPE total_file_count gauge
total_file_count <%= stats["total_file_count"] %>
# TYPE last_snapshot counter
last_snapshot <%= Date.parse(snapshots[-1]["time"]).to_time.to_f %>

We can invoke this template using something like the following incantation -

ruby -r json -r erb -r date -e 'ERB.new($<.read).run' template.erb

so to tie it into the rest of the system and make it run when the backup completes, I call it from the restic backupCleanupCommand.

TIL writeShellApplication which is like writeShellScriptBin but sets up the PATH and has a few other niceties.

# modules/backup/default.nix
{ config, pkgs, ... }:
let
  inherit (pkgs) writeShellApplication;
  passwordFile = "/etc/nixos/secrets/restic-password";
  repository = "/var/spool/backup";
  template = ./template.erb ;
  prom_export = writeShellApplication {
    name = "prom_export";
    runtimeInputs = with pkgs; [ restic ruby ];
    text =''
       mkdir -p /var/lib/prometheus-node-exporter-text-files
       chmod 0775 /var/lib/prometheus-node-exporter-text-files
       RESTIC_PASSWORD=${passwordFile} \
       RESTIC_REPO=${repository} \
       ruby -r json -r erb -r date -e 'ERB.new(File.read(ARGV[0])).run' ${template} \
         > /var/lib/prometheus-node-exporter-text-files/restic.prom.next
       mv  /var/lib/prometheus-node-exporter-text-files/restic.prom.next \
         /var/lib/prometheus-node-exporter-text-files/restic.prom
    '';
  };
in {
  config.services.restic.backups = {
    local =  {
      # your restic config here
      backupCleanupCommand = "${prom_export}/bin/prom_export";
    };
  };
  # etc etc
}