Hard pass#

Sat, 14 Aug 2021 21:31:26 +0000

I've got the key
I've got the secret
– From the Urban Cookie Collective's guide to password management

For reasons that seemed good at the time, I've written a password manager. It's a lot like pass ("the standard unix password manager") - which I have been using up 'til now - but it uses age instead of GPG to do the heavy lifting.

moss, the Maybe-Ok Secrets Store, is a 400-line Ruby script that uses only libraries provided by a default Ruby installation, plus 520 lines of testing code (Cucumber and RSpec).

Some random observations follow:

almost all of it was test-driven, but the tests are for the most part end-to-end Cucumber tests that run the script in a subshell instead of calling classes/functions/methods directly. I was not expecting to get that far with end-to-end tests, but when all's said and done it's a pretty small system.
There are certain deficiencies in the Cucumber step definition language that get more annoying as the test suite grows. Most notably, the use of ad hoc @var to pass state from one step to the next. I don't have a solution here, I'm just complaining about the problem.
I don't claim to be a security expert, which might make me not a good person to attempt this kind of project. I have carefully avoided rolling my own crypto by using a third party program for that, and I am quite pleased with my use of the various affordances of Kernel.system, IO.popen, Kernel.spawn and Tempfile to avoid even passing plaintext or passphrases in or out of the Ruby process. There's a lot of flexibility in that family of methods if you read the docs. More information about my security-related design choices in the relevant section of the README
I am simultaneously proud and ashamed of the command line parser I created by using various metaprogramming features in ways that may or may not be generally recommended. (Don't try this at work, kids - at least, not if you work for my employer, metaprogramming in company code is firmly in the "presumption of bad" bucket).
I didn't know, but now I do, that there is support in the Ruby standard library ('Fiddle') for calling C libraries without having to write extensions and needing a C compiler. I didn't actually need to - I thought I was going to need mlock(2) before I realised it wouldn't help and found a way to use a temporary file instead - but it's nice to know it's there.
It's a single script with no dependencies other than a Ruby installation, and this is intentional: it means you can easily install it anywhere just by copying the script. I might reconsider the "single script" constraint if/when it passes 500 lines. I am less likely to renege on "no external dependencies", because I've too often had trouble running programs that require Bundler from inside a project with its own Gemfile.

It's been a long time since I wrote more than about 5 lines of Ruby for anything outside of a work context: for 'fun' projects I tend to pick languages which I don't get a chance to use 9-5. Ruby for this task was definitely less than awful, though.

Slightly too much Kodi for fun#

Wed, 04 Aug 2021 22:02:24 +0000

I'd been vacillating for a while about buying a new monitor, but eventually I pulled the hammer (is that the idiom?) on a spangly new Dell S2721SQ, which arrived yesterday and provided the incentive to look at NixElec again. Because it (the monitor) has speakers, which means I have the hardware to fix the audio issues without having to commandeer the family TV.

Second rate

I don't claim to understand how ALSA works, and Kodi's approach to ALSA is even more weird, but I did eventually make it work for 44.1kHz sources: define an ALSA fixed-rate pcm for Kodi that is hardcoded to S16_LE format, and then tell Kodi about it in advancedsettings.xml

A sticky GUI mess

To the extent that Kodi can be configured through files, they're XML files. There is a toXML builtin in Nix, but it only generates a particular XML representation that would need XSLT to turn into files that Kodi likes - and XSLT for me is assigned firmly to into the same "tried it once, not going back to that" bucket as are m4 and Java applet programming.

What I really wanted is something that would let me write out (or generate!) a nested attrset describing the structure I want, and turn it, possibly via JSON, into XML. Python's dict2xml is very nearly it, but has no support for XML attributes, so I had to invent something slightly more complicated.

Sadly, the extent that Kodi can be configured through files is not the full extent. Although the sources are defined in XML, the content of each source (tv shows? movies? music?) seems to be set in a Sqlite database, which is another level of complexity to manage. So there's still manual twattery on the GUI to deal with.

Just enough NixOS for Kodi#

Sat, 26 Jun 2021 21:56:17 +0000

I've had an Odroid C2 sitting under the TV for a a year or so, mostly used for playing the Shaun the Sheep videos that live on my PC upstairs. I put LibreELEC on it when I bought it, and subsequntly tweaked it in a succession of ways that I basically don't remember.

Quite recently I decided that I didn't need this pocket of divergence in my otherwise mostly congruent domestic computing infrastructure, so I set about installing NixOS on it. There's a description in the NixOS Wiki of how to do this which is a good starting point, but there was some other stuff I had to figure out.

Caveat

I still don't have everything figured out. In particular, it seems to want to play audio slightly too fast, I think because somehow it has decided that my 48kHz sound device will accept 44.1kHz PCM without need of resampling. I probably won't update this blog entry when I figure that out, but I will update the repo it points to.

I would caution against following these notes from start to end and doing everything manually as I discovered it, because there's a configuration.nix at the end that has most of it automated.

Initial install

The board has some weird requirement to install a binary blob and an u-boot image in a very specific part of the storage medium. "Note this assumes u-boot is in partition 1 of your board's connected eMMC", say the instructions. I am happy to report that the same merry dance works just fine on an SD card if like me you were too stingy (or poor) to spring for the eMMC.

It needs to use a traditional MBR partition scheme, not GPT
I found that I needed to run the sd_fusing script again after each time I touched the partition table

Kernel rebuild

Although the generic NixOS aarch64 kernel will work to get you to a login prompt, it doesn't work well for video playback. This is because there's a meson_vdec module needed for hardware video decoding that isn't in it. After some experimenting I came up with the following configuration snippet

  nixpkgs.overlays = [
    (self: super: {
      linuxPackages = super.linuxPackages_latest.extend (lpself: lpsuper: {
        kernel = super.linuxPackages_latest.kernel.override {
          extraConfig = ''
             STAGING y
             STAGING_MEDIA y
             VIDEO_MESON_VDEC m
          '';
        };
      });
    })]

but this is where I found that the machine is slow as molasses at compiling - although to be fair the default NixOS aarch64 kernel build is a huge task on any hardware due to the large number of modules it builds.

Hot cross bins

So, time to figure out how to cross-compile it on an x86_64, and given that I want this to be repeatable, how to cross-compile the entire system instead of cross-building bits and native-building other bits and copying artifacts around by hand. Like this, is the short answer (adjust pathnames/hostnames as needed)

 # build the system
 NIXOS_CONFIG=/home/dan/src/odroid/nixelec/configuration.nix \
  nix-build -E 'let pkgs = (import /home/dan/src/nixpkgs) {};
  in (pkgs.pkgsCross.aarch64-multiplatform.nixos
      /home/dan/src/odroid/nixelec/configuration.nix)'.config.system.build.toplevel

 # copy it to the target device
 nix-copy-closure --to root@odroid.lan -v --include-outputs \
   ./result && ssh root@odroid.lan \
   `readlink result`/bin/switch-to-configuration switch

As usual with cross-compilation, this unearthed a bunch of packages that don't cross-compile because nobody really understands when to use buildInputs vs nativeBuildInputs (I'm projecting here, it might just be me), one package that doesn't cross-compile because it's magic - I refer of course to gobject-introspection - and some packages that need their derivations tweaking so that they don't depend on any of the other stuff that depends transitively on gobject-introspection.

Image problems

Careful readers will observe that the shell incantations above are predicated on having a running NixOS Odroid system already that you can ssh into - so, how do you get that in the first place? I augmented the configuration further so that it can also be used to produce an SD card image which has builtin the faffage needed to get the firmware and U-boot binaries injected at the right offsets. To generate this, we do

NIXOS_CONFIG=/home/dan/src/odroid/nixelec/configuration.nix \
 nix-build -E 'let pkgs = (import /home/dan/src/nixpkgs) {};
  in (pkgs.pkgsCross.aarch64-multiplatform.nixos
      /home/dan/src/odroid/cross/configuration.nix)'.config.system.build.sdImage

(this is very similar to the previous command except now we're building sdImage instead of toplevel)

and then find the output in result/sd-image/nixos-sd-image-21.11pre-git-aarch64-linux.img - again, Your Pathnames May Vary. dd this to whatever device corresponds to the SD card you plan to insert into the Odroid machine and you should be good to go. It should install a valid SSH key for the root user, but it would be as well to check.

Where is it?

https://github.com/telent/nixelec

What's left to do?

make the sound work better
more declarative config for Kodi itself, instead of configuring it through the UI
push the cross-compile fixes upstream
find out why it hangs when I ask it to reboot and fix that

What's up, D?#

Fri, 30 Apr 2021 17:27:22 +0000

One of the nice things about side projects (as opposed to the ones I get paid for) is not having to be accountable to anybody for timely delivery, and the freedom to head off on tangents or side investigations. I have recently picked up NixWRT again, but I regret to inform you that "off piste" is where I am right now.

Backstory: monit is not a great service monitor for my needs, mostly because it isn't the parent process of the service daemons it starts, so can't tell when they exit without polling/pid files etc - which is slow and unreliable. So I decided that the world does not have enough init replacements already, and because of the risk that anyone else might otherwise find it useful I decided to write it in Fennel. Then a little while later, I decided to do something else for about five months, and now I have no mental context of what I was doing.

So why not start again? This time it's called upd and although it shares some ideas with swarm, I am attempting to use tests to drive more of its design.

I started by sketching out the shape of a plausible service monitor for a pppoe daemon. You can see that the state logic is complicated and it has many collaborators, so I install mock versions of them by setting entries in the package.loaded table which require checks before loading a file. The most complicated bit is probably the fake event loop: the test setup provides an array my-events of functions which may update state, replace mocks etc, perform assertions etc, and and the mock event loop runs them one by one. This lets me write my first test, to see that the daemon is started when the interface is present. A followup refactor replaces the longwinded calls to tset package.loaded with more intention-revealing names mock and mocks.

The second test checks that we observe an exponential backoff when the process fails to start. We don't have any visibility into the actual backoff state in the system under test, so have to observe from the outside. First we stop the process from an event function, and then poll the process state repeatedly while incrementing a counter until it restarts. 0e966459

This is convoluted and might suggest that we should make that state visible. But it also prompted me to try making backoff state be a property of the process monitor itself, not the overall system under test - suppose we want to write a script that watches two or more processes? The next few commits

move the backoff state variables into the process
make the process own backoff state mutation by adding a backoff method
move backoff testing into the process with a backoff-expired? method

Of course, making the backoff state a property of the process has the side effect of making it visible. The next test is that the process is stopped if the underlying link is lost, and it's actually quite handy to be able to reset the backoff interval in the test setup: 0350c8aed6a

Something was still bothering me here, though. In order to move the backoff testing from the script to the process, I'd done

-      (when (and (not (process.running? pppd))
-                 (nil? pppd.backoff-until))
+      (when (not (process.running? pppd))
         (pppd:backoff))

This is logically correct, as pppd:backoff does nothing anyway if pppd.backoff-until is nil. But it's really quite non-obvious. What we actually want to say here is "if the process died, back off" - so let's say it! Add a process.died? method and rewrite the test to use it.

The last interesting change in this sequence was triggered by looking at the event loop, when I realised that we don't anywhere use the value of the event that comes back from next-event - we wait for something to happen, but then we discern what happened by testing various bits of state. To say the same thing again in fewer words: we're don't receive events, we wait for state changes. So let's change the code to use better names

What's the conclusion? Is there a conclusion? A few thoughts -

for the TDD purists, it wasn't "real" TDD because I wrote quite a lot of implementation code first instead of writing the test first.
by listening to the pain of testing I was able to improve the micro-level design in a few aspects: I am happier with the result than with the initial code
by having the tests I was able to make those changes while being reasonably confident that I hadn't broken anything
the next thing on my list (which is in progress already) is writing the service monitor for a hardware ethernet device. When it's in slightly better shape I will write another post full of netlink and qemu trivia, but until then, goodbye.

First steps in Colemak#

Thu, 15 Apr 2021 14:29:26 +0000

This is more of a placeholder than anything else, but I decided at the weekend that it was time to learn to touchtype, and further, that it would make sense to switch to Colemak at the same time. I don't yet know if it actually will prove correct, it's too early too say.

I'm using Colemak Academy and have achieved 97% on the home keys at an astonishingly speedy 18 words per minute, but the top and bottom rows are still strangers to me. And only after trying to use Emacs did I learn that the Colemak layout rebound Caps lock (which I bind to Ctrl) as Backspace).