diary at Telent Netowrks

Building Clojure projects with Nix#

Sat, 19 Aug 2017 23:23:07 +0000

Way back in May, or "the previous entry" as we like to call it in Telent Blog Time, I said

[using mvn2nix] doesn't actually work for Clojure/Leiningen projects. Look out for another blog entry soon that describes a whole completely different approach you can use there.

and then it turned out not to work, and then I thought I'd found a different approach, and then I got distracted, and then $WORK Happened, and, you know, Events. But here is an account of one way you can do it (at least, It Worked For Me) and some things I think I remember about the dead-ends I hit along the way.

Starting with the conclusion

Ingredients:

Try it. Send feedback

Why Leiningen didn't work

What we planned to do with Leiningen was not completely dissimilar. At "release time" we were going to

then at "deploy time",

The trouble with that was that Leiningen appears to want to do dependency resolution all over again whenever it's run, which means that we had to download not only the required jar but in some cases one or more older versions too, just so it can persuade itself it has the right versions.

Kick me harder

Then I tried converting it to use boot. On the face of it this should be much simpler because boot has a rather lovely with-cp task which lets one explicitly specify the CLASSPATH. In practice, though, boot itself wants to auto-update and install stuff in $HOME which is a problem for NixOS autobuilders which run in a context where $HOME is set to a non-existent directory.

I'm still using boot in development and to provide the scaffolding for the bit of code that creates the json file (I made it a boot task), but I can't use it to do the production build.

Next up, use NixOps to creat the rest of the machine on which the app is going to run

Building Maven packages with Nix#

Wed, 10 May 2017 01:06:07 +0000

[ Note to Clojurists using Leiningen: I found, towards the end of this maven2nix process, that the pom.xml files that lein pom creates are not actually sufficient to the task of building jars. If that's why you're reading this, this may not be the right thing to be reading. ]

Nixpkgs has some tools for building Maven projects but (as far as I can determine) absolutely no examples of their use. It took me some time to figure things out from trawling through mailing lists, not least because my Nix-fu is not (yet?) up to reading the buildMaven source and shell script and saying "hey, that's obvious". I write these notes in the hope that eddies in the space-time continuum will cause them to pop up a week ago when I needed them, but failing that, maybe they'll be useful to someone else who struggles with it.

If you have a Java project whose dependencies are managed by Maven, you will find that building a Nix derivation for it is non-trivial, because maven expects to be able to figure out the dependencies and download them as it runs. Nix doesn't appreciate this, because derivations should be pure functions of their inputs and not depend on what happens to be hosted on random web sites at any given time. Indeed, proper "pure" builds are run in a context that doesn't even allow them outbound network access.

Other languages with similar package dependency managers have a similar problem: for example, Ruby projects that use Bundler, or Node projects that use npm/yarn. The solution is also similar: we break it into two parts. First, generate a list of all the dependencies - including the transitive dependencies - so that Nix can download them into the store; second, run the build in "offline" mode and tell it to find the dependencies in the store instead of on the Internet.

Get the package list

My overpowering impression of Maven based on having used it for three days is that it is entirely composed of plugins. Great plugins have little plugins upon their backs to bite 'em, and little plugins have lesser plugins - and so, ad infinitum. In this case, the plugin is called org.nixos.mvn2nix:mvn2nix-maven-plugin and it provides a command called mvn2nix. In principle using this is a simple matter of writing (or somehow obtaining) a pom.xml file, then doing

$ mvn org.nixos.mvn2nix:mvn2nix-maven-plugin:mvn2nix # don't do this

to generate a file called project-info.json. However, in practice there are a couple of issues:

The issue is that mvn2mix generates different output for dependencies that already exist in your local repository than it does for packages it fetches, and the workaround (as described in the issue) is to run it against an empty directory:

$ mvn  -Dmaven.repo.local=$(mktemp -d -t maven)  org.nixos.mvn2nix:mvn2nix-maven-plugin:mvn2nix # mmm


"https://repo1.maven.org/maven2//logkit/logkit/1.0.1/logkit-1.0.1.pom"
                                ^ non-ideal

This is the kind of thing for which sed would be the unix-philosophy-solution, if you can handle the resulting leaning toothpick syndrome. Or you could just use Perl

$ cat project-info.json | jq . | \
 perl -pe 's,https://repo1.maven.org/maven2//,https://repo.maven.org/maven2/,g' > $$ \
     && mv $$ project-info.json # or something

This probably also constitutes a bug but I've not yet taken the time to find out if it's with mvn2nix or with my generated POM, or whether it's been reported previously. Anyway, achievement unlocked.

Writing the Nix expression.

I've done this as a standalone expression that you can drop into your project as default.nix and run nix-build on: if you were writing something to add to the packages set in the nixpkgs repo then I think the preamble would look different (caveat: don't understand, haven't tried it).

with import  {};
let jar = buildMaven ./app/project-info.json;
in stdenv.mkDerivation rec {
  buildInputs = [ makeWrapper ];
  name = "my-app";
  version = "0.1.0";
  builder = writeText "install.sh" ''
    source $stdenv/setup
    mkdir -p $out/share/java $out/bin
    cp ${jar.build} $out/share/java/
    makeWrapper ${jre_headless}/bin/java $out/bin/${name} \
      --add-flags "-jar $out/share/java/${jar.build}"
  '';
}

Some points of note

I'm not a Nix expert (did I say that already?). Where this is non-idiomatic it's more likely because I don't understand the idiom than through any kind of conscious decision process.

Coming soon

As I said at the start, this doesn't actually work for Clojure/Leiningen projects. Look out for another blog entry soon that describes a whole other completely different approach you can use there.

NixOS on Linode KVM#

Mon, 19 Dec 2016 21:28:08 +0000

This is much easier than you'd think by Googling - but still complicated enough to be worth writing down. Note that this description is rather on the terse side, and probably more understandable if you've done a NixOS install on "regular" hardware previously.

1) create a new Linode instance

2) select it in Linode Manager

3) make some disks: for each of these, go to 'Create a new Disk' and fill in the appropriate fields

4) now go to 'create a new configuration profile'

5) before you turn it on, go to the Settings tab and disable Lassie - otherwise it just gets confusing having it reboot when you're not expecting

6) first time boot will be into the Rescue system, so go to the Rescue tab and click 'Reboot into Rescue Mode'

7) now connect to it using lish. You should be shown a typical boot sequence ending in something like

Welcome to Finnix!
[*] Total memory: 1998MiB, shared ramdisk: 1543MiB
[*] Finnix media found at sdh
[*] System: Intel Xeon E5-2680 v3 2.50GHz
[*] Running Linux kernel 4.1.2-finnix on x86_64
[*] Finnix 111 ready; 464 packages, 146M compressed
root@ttyS0:~#

8) now it's time to download the ISO. From the Getting NixOS page, find the URL for the latest 64 bit Minimal Installation CD and then run something rather like

root@ttyS0:~# curl -k https://d3g5gsiof5omrk.cloudfront.net/nixos/16.09/nixos-16.09.1272.81428dd/nixos-minimal-16.09.1272.81428dd-x86_64-linux.iso |dd bs=1M of=/dev/sda
root@ttyS0:~# sync
root@ttyS0:~# halt

9) Now we can boot into the NixOS installer. Check that you still have lish running in a terminal window (reconnect to it if not), then click 'Boot' in the Dashboard. You should see a Grub menu appear: cursor down to the 'NixOS ... without modeset' option then hit TAB to edit the command. Append console=ttyS0, then hit RETURN to boot. You get a NixOS boot sequence in lish, then 

[root@nixos:~]#

10) This is a good time to bring up the NixOS manual in a browser tab, especially if you can't remember what you did last time

11) Create a partition table on /dev/sdb and a primary partition that fills it, then put a filesystem on it:

[root@nixos:~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.28.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device /dev/sdb already contains a ext4 signature.
The signature will be removed by a write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x01a28df9.
Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-2097151, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-2097151, default 2097151):
Created a new partition 1 of type 'Linux' and of size 1023 MiB.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
[  441.705223]  sdb: sdb1
Syncing disks.
[root@nixos:~]# mkfs -t ext4 /dev/sdb1

(This looks verbose but it's almost all default options - what I actually typed to produce that output was n, RETURN, RETURN, RETURN, RETURN, w, Control-D)

12) mount the disks 

[root@nixos:~]# mount /dev/sdc /mnt
[root@nixos:~]# mkdir /mnt/boot
[root@nixos:~]# mount /dev/sdb1 /mnt/boot

13) now you can run nixos-generate-config --root /mnt. Edit the generated /mnt/etc/nixos/configuration.nix and uncomment/amend/check the following items

14) and run 

[root@nixos:~]# nixos-install    # stuff happens, at length
[root@nixos:~]# umount /mnt/boot
[root@nixos:~]# umount /mnt

15) now it's time to try booting from the "hard disk": go back to the Linode Manager and create another Configuration Profile

Select the new profile in Dashboard and hit 'Reboot'. Eventually, you will see the very welcome Welcome to NixOS banner and be presented with a login prompt.

16) Login as root, using the password you set duing nixos-install. Check you have network connectivity. Arrange credentials (password or ssh key or whatever) for whatever non-root user(s) you put in configuration.nix. Try ssh into the box. When you're happy it works, use C-a d to exit lish. Your work here is done.

Hotplug scripts in NixOS#

Thu, 15 Dec 2016 08:46:21 +0000

Prompted partly by the new Met police online traffic incident reporting tool (but it's been something i've been thinking about for a while) I bought an action camera the other day: it's a "Savfy", which is a cheap clone of the SJ4000 (which is itself a cheap clone of a Gopro).

Since I need to plug it into a USB port every day or two to recharge (battery life is a claimed 1.5 hours, haven't tested this yet) I thought it would be good to automate downloading the data off it as well. This is almost my first foray into systemd and udev, and certainly the first time I've tried it in NixOS, so I have reproduced my findings below.

First, we need to recognise when the camera is connected. It's USB mass storage, which means it shows up as a SCSI disk device (e.g. sdb). In /etc/nixos/configuration.nix we add a stanza something like this:

  services.udev = {
    path = [ "/home/dan/udev/bin" ];
    extraRules = ''
    ACTION=="add", KERNEL=="sd*[0-9]", ENV{ID_SERIAL}=="NOVATEKN_vt-DSC*", RUN+="${pkgs.systemd}/bin/systemctl --no-block start copyCamFiles@%k.service"
    '' ;
  };

There are a few things worth noting here.

Great. We've got the trigger, where's the service? Again in configuration.nix

  systemd.services."copyCamFiles@" = {
    bindsTo = [ "dev-%i.device"] ;
    environment = { 
        RSYNC = "${pkgs.rsync}/bin/rsync"; 
        MOUNT = "${pkgs.utillinux}/bin/mount"; 
        UMOUNT = "${pkgs.utillinux}/bin/umount"; 
    };
    serviceConfig = {
        Type = "simple";
        ExecStart = "${pkgs.bash}/bin/bash /home/dan/udev/bin/cp-actioncam.sh %I";
    };
  };

The @-sign in the service name means this isn't actually a service, it's a template (i.e. we can pass parameters to it to instantiate services). When it's invoked by udev, the parameter passed will be the device name of the newly-added partition. We export the pathnames of some utilities that the script will need, because I haven't built a nixos derivation for the script itself. simple as a service type means (I hope) that it's not started unless asked for and that nothing is going to try to restart it when it terminates.

Finally, here's the cp-actioncam.sh script

#!/usr/bin/env bash
DEVNAME=$1
MP=/run/tmpmounts/$$/
set -e
unmount() {
    $UMOUNT $MP;
    rmdir $MP;
}
trap unmount 0
OUT=/home/dan/Videos/actioncam
mkdir -p $MP $OUT 
$MOUNT /dev/$DEVNAME $MP  
$RSYNC -a $MP $OUT

And there you have it. There are a bunch of refinements that could profitably be made: most obviously, notifying the user somehow when the copying is finished and the device may be unplugged, and not putting root-owned files into a non-root-users home directory. But this will do for now.

Some browser tabs I can now close:

Huawei E3372 with OpenWRT#

Sat, 12 Nov 2016 22:25:34 +0000

I've just moved house. The new house has no broadband or even phone line, and when I last asked about the progress of getting a line connected I was told that "Your order is currently in the newsites stage and the offsite ducting has started" and it was going to take most of a month before I could even start ordering broadband.

Which means I need mobile broadband stuff. I looked on Ebay for USB LTE sticks, and I looked on comparison sites for cheap data SIM-only deals, and ended up with a Huawei E3372h. To make this work with OpenWRT took a couple of evenings swearing at it, so here's the end result: note that all of these things were for me sufficient but may not be to you necessary.

chat
comgt
comgt-ncm
dhcpcd
kernel
kmod-mii
kmod-usb-net
kmod-usb-net-cdc-ether
kmod-usb-net-cdc-ncm
kmod-usb-net-cdc-subset
kmod-usb-net-huawei-cdc-ncm
kmod-usb-serial
kmod-usb-serial-option
kmod-usb-serial-wwan
kmod-usb-wdm
libpthread
librt
libusb-1.0
libusb-compat
usb-modeswitch
usbutils
wwan
zlib


config interface 'WAN'
        option proto 'ncm'
        option apn 'everywhere'
        option ifname 'wwan0'
        option pdptype 'IP'
        option device '/dev/ttyUSB0'
        option delay 15

I get about 12-17Mb/s (over wireless, sitting next to the router) according to fast.com which is not superfast considering the hardware is supposed to be capable of 150Mb/s and the package I'm paying for is "up to 60Mb/s". There may still be something I'm missing in the configuration. Whether I spend much more time looking at it is going to depend mostly on how much faff it takes to get proper broadband installed. It might be a while