MuDDLe, a faster simpler maildir downloader#

Sun, 02 Feb 2014 23:27:24 +0000

One of the services running on my old Bytemark VM was the Dovecot IMAP server. When I started thinking about configuring it on the replacement box I realised that I don't actually need IMAP these days, so, er, why bother? So I didn't. But what I do need is a way of getting the Maildir on that machine onto other machines, and everything I looked at to do this job (other than rsync) was fearsomely complicated because it also catered for a zillion other file formats and/or transports. Or because it wanted to sync in both directions, which I don't really care about that much.

So, muddle

In essence, what it does is this: connect to the remote host, find list of files in cur/ and new/, compare with similar list on local host, create tar stream of differences, transfer it, unpack each transferred file into tmp/ and atomically rename into cur/ when done.

This adds up to one transfer of file names and a second transfer of all the file contents. Each of these is one-way and distinctly non-chatty, so it should have reasonably good network performance characteristics and you can use ssh compression if your network bête noire is bandwidth itself and not just latency.

It might not be as simple as possible, but at the same time it might also be simpler. For example, and as alluded to above, the download is one-way only, so it won't e.g. update the server to mark messages as read. If you care about that stuff, this is not for you.

(Why not rsync? It can't detect that a rename from new/foo to cur/foo:2, is a rename, so treats the latter as a new file. Which is a teensy bit suboptimal)

Keeping secrets in public with puppet#

Mon, 10 Feb 2014 18:26:14 +0000

I recently stumbled across dotgpg , which is in essence some scripts to make it easy to securely keep secret files in a public git repo, which are protected by means of having been encrypted (usually for multiple recipients). It comes with capistrano glue for decrypting them again and sending them to your production servers, but it doesn't quite fit my masterless use case where everything happens on the same box and there isn't the same notion of a 'target system'

But it set me to thinking: what if there was some kind of gpg agent that let you type in your key once and used it for multiple decryptions (turns out there is) and what if you then wrote some custom puppet function, let's call it decrypt, so you could then say

  file {'/etc/wpa_supplicant.conf':
    content=>decrypt("templates/etc/wpa_supplicant.conf.gpg"),
    owner=>root,
    mode=>0600
  }

and everything would Just Work. Well, turns out I did and you can and (as far as I can tell) it does. The custom function is as simple as creating the file puppet/parser/functions/decrypt.rb inside /etc/puppet (or wherever) containing

module Puppet::Parser::Functions
  newfunction(:decrypt, :type=>:rvalue) do |args|
    filename = args[0]
    `/usr/bin/gpg --use-agent --decrypt #{filename}`
  end
end

and now at the expense of a slightly more convoluted puppet invocation

$ sudo  make -C/etc/puppet/ GNUPGHOME=$HOME/.gnupg GPG_AGENT_INFO=$GPG_AGENT_INFO

I can put my wpa network configuration (and my jabber passwords, and smtp client passwords, and some other stuff I can't right now remember what it is but am sure exists) alongside my all my other configuration instead of either having to do something silly with git submodules or rebuilding it by hand. Am now furiously trying to memorise my passphrase.

Better error checking would be nice, so that it doesn't overwrite a perfectly good config file with an empty one if the gpg stars aren't all aligned, but that is left as an exercise for next time.

If you can see this, it worked#

Thu, 13 Feb 2014 22:50:59 +0000

For values of "this" which you don't care about and can't see, but my hacky homebrew blogging engine now watches a bare git repo using inotify and runs a checkout/refreshes its content when it sees changes. I'm not sure it wouldn't have been simpler just to make it die and then use a, y'know, shell script or something to run a git pull before restarting the server, but I did it this way because ZERO DOWNTIME.

Anyway. You don't see and can't care, or possibly vice versa. But the previous Heath Robinson stuff with git hooks wasn't working now that the bare git repo is owned by someone other than the uid that runs the http daemon, so a different Heath was called for.