Migrating Xen to KVM#

Wed Jun 3 15:06:29 2009

Until very recently I ran this blog site, and a couple of others (test sites for work, that kind of thing) in a Xen virtual host on the pc sitting on my desk. While it worked ok for that purpose, it really didn't coexist nicely with much else: the Xen kernels available in debian have a showstopper bug for running X11, are missing modules for e.g. cpu temperature monitoring, etc. And the process to build ones own is not usefully documented. (I apt-get sourced my kernel image, and it downloaded a completely different kernel version). I don't think the stuff is unusually buggy, it probably just doesn't have enough eyeballs on it to find all the weird interactions with other components or to patch them when they do turn up.

So when my four year old Athlon motherboard died recently, the opportunity to rejoin the mainstream and start using kvm was the silver lining in the "ew, hardware" cloud.

A hard ware is gonna fall

I decided to go Intel this time despite my AMD-supporting underdog instincts, chiefly because I figured that (a) every Intel cpu on the planet should have VT support by now, (b) onboard graphics are well-supported in xorg, (c) everything I read about the E5300 CPU said it had very good price/performance ratio and overclocking it someday sounded like a fun project. So basically I bought that and the first G45 board I found that had onboard graphics (this is the distinction between G45 and P45 which doesn't), three SATA ports and was made by someone I'd heard of. It's the Asus P5Q-EM, and it apparently also is a good one for overclocking.

After putting it all together, I found that it didn't work. Important lesson here: for 'market segmentation' reasons - and I was certainly pretty cut up about it, though that may be not the kind of segmentation they mean - Intel only implement the necessary VT bits on CPUs that cost north of £100, not on the cheap and lovely E5x00 series even though they have the (arguably equally "enterprise niche") 64 bit support. So, gah. So I ended up with an E8200 as well, and a spare cpu+fan that is probably going on Ebay soon.

This walk is made for booting

KVM now runs, which is a start. Note that you need the kvm module and the kvm-intel module, or it will complain that you have no CPU support. You may also have to fiddle with your bios settings, although I didn't.

To convert a Xen disk image into an image that kvm understands is still a bit fiddly: in brief, xen images are filesystem (= partition) images, whereas kvm wants entire disks and they need to be bootable (i.e. contain grub and a kernel). Here's how I did it, prefaced by the obligatory "it worked for me, but it's not my fault if it doesn't for you. Back up first" caveat

Obligatory caveat: it worked for me, but it's not my fault if it doesn't for you. Back up first.

1) Identify your disk image. On a Debian box it will be in some place like /usr/local/lib/xen/domains/vitrual.4a.telent.net/disk.img

# cd /usr/local/lib/xen/domains/no.stargreen.com
# file disk.img 
disk.img: Linux rev 1.0 ext3 filesystem data, UUID=e6143cef-c3ed-4c21-94d9-32d4ba186910 (large files)
# ls -l disk.img 
-rw-r--r-- 1 root staff 4294967296 2009-06-03 14:33 disk.img

Looking promising. So let's create a qemu image.

# dd if=/dev/zero of=mbr bs=8225280 count=1
# cat mbr disk.img mbr | cp --sparse=always /dev/stdin qemudisk.raw

Why these numbers? It's a throwback to ancient technology: ye olde PC accessed disks using cylinder/head/sector technology, and the upper bounds on each bit are 16383, 16, 63. In my testing, KVM ignores the CHS settings in the disk image and assumes 16 heads/63 sectors, so we're going to make it easy for ourselves by adopting that geometry ourselves, and since this makes each cylinder 8225280 bytes long we'll stick that much blank space on the front of the image for MBR/partition table/etc so the first partition is at cylinder 1

And the additional space on the end? In my first few attempts I ran into trouble with e2fsck complaining that the filesystem was bigger than the disk image. This is, of course, impossible, and I imagine was due to some kind of rounding error, but padding it out this way is easier than investigating.

The sparse flag to cp might save us a bit of disk space if we have any great big expanses of zero in the image. Mine didn't, as it turned out, but YMMV.

Next up, partitioning. sfdisk accepts input on stdin and uses it to destroy your data, which makes it in some ways the perfect Unix-philosophy tool. GNU Parted will never give you this kind of buzz, so be very careful here.

 # echo '1' | sfdisk qemudisk.raw  # DO NOT MISTYPE THIS
 # sfdisk -l qemudisk.raw 
 Disk qemudisk.raw: cannot get geometry

 Disk qemudisk.raw: 524 cylinders, 255 heads, 63 sectors/track
 Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
 qemudisk.raw1          1     523     523    4200997+  83  Linux
 qemudisk.raw2          0       -       0          0    0  Empty
 qemudisk.raw3          0       -       0          0    0  Empty
 qemudisk.raw4          0       -       0          0    0  Empty

Android G1 at six months#

Mon Jun 15 13:46:04 2009

The trouble with doing long-term reviews of mobile phones is that by the time you've had them anything approaching a long time, they've been superseded already and are probably about to be discontinued.

But anyway. I've owned this thing since shortly after they came out and having recently finally peeled off the increasingly tatty piece of cellophane it shipped with to protect the screen, I reckon it's time for an update.

The hardware

I won't go into detail about the hardware, as the software aspects are going to be more interesting long-term. The summary is "good but not excellent": it has not bent, broken, warped, wefted, spindled, folded or self-mutilated, but on the other hand its styling is still closer to a brick than to an Iphone. Apart from the battery, which deserves its own section, the single stupidest hardware feature is the plastic plug which is supposed to cover the mini-usb charging/headphone socket. Second stupidest is probably the socket itself: although mini-usb is the only sensible design for power and data connections, the adaptor necessary to plug normal audio headphones in is wickedly fragile and can be had only from chinese ebay sellers

I said the battery deserves its own section. Here it comes:

Battery life is crap. Recharge every night, because it will not last a second day

Message ends. I know it's an incredibly versatile device that's more like an ultraportable computer than a mobile phone as we know them, and also that by disabling all the interesting features I could probably get it to last two days, maybe, at a push, but that would kind of negate the point of having paid for the extra features in the first place, now wouldn't it?

The software

It's now six months after launch and I have to concede that despite having upgraded to the "Cupcake" OS version 1.5, the software still feels unfinished in some areas. The Google contact/calendar integration is a win, but it still lacks certain things that an ordinary phone can do without problems, like selecting a phonebook entry and beaming it or texting it to a friend. And some of the things it does do give the strong impression of never having been tested on real people. The Mail app at launch was famously bad enough that a third-party group of hackers forked it just to get it to a usable state (their fork, K-9, has since come on in leaps and bounds and I strongly recommend it for non-Gmail IMAP users). My particular ire is reserved for the calendar, which defaults to presenting a view of the month with coloured blobs to indicate when things are happening but no clue as to what those things are. And the camera - I don't know whether to blame hardware or software or both, but I could probably paint the scene in less time than it takes to focus, and with a better result in any setting less well-lit than an operating theatre.

This sounds pretty negative and it's mostly not. It's just that when you do get smacked by the 10% stupidity sections after the 90% (in most cases) that is there has lulled you into quiet babylike stuporific contentment, it smarts twice as much. My recommended apps, for the record:

• K-9 (unless you use Gmail)
• ConnectBot: an ssh client
• Twidroid: still the best twitter client I've seen
• Greed: RSS reader syncs with Google Reader, so you can discover feeds and manage subscriptions on your desktop instead of having to type the URLs in on the device keyboard
• Ringdroid: MP3 editor and ring-tone creator

Development

The phone this replaces is a Sony-Ericcson M600. The PDA it replaces is a Palm TX. The first thing to say about it, then, is that for openness it's light-years ahead of either. Reasonably well documented, easy to get into (no need to join developer programmes or send your apps away to be certified) and there's a linux-hosted toolchain. I haven't explored it extensively but I did without too much trouble write half an app that logged gpx tracks (but then My Tracks came along and obsoleted my effort), and poked around inside K-9 a bit. It's Java (practically speaking: legally it's merely Java-like) which language I grant you nobody ever expanded their minds by learning, but on the other hand has never caused anyone's brain to leak out through their ears in quite the same way as C++ or PL/SQL can.

That sounds disturbingly - albeit quite faintly - like praise, so I have to add that the second thing to say is that they've tragically missed the point: it's still an ocean removed from hackable in any useful sense. All the support is there to write your own apps is no use to you if you merely want to modify someone else's app, and even if you are starting with one of the open-source apps you still have the burden of maintaining your own private fork just to support your private hacks.

It's not all bad: the use of Intents should make it possible to "deep-link" into other people's apps (see Oilcan for example), but it's still more work than, say, Emacs hacking or Amiga ReXX ever was.

Hacking

Underneath it all, as I'm sure everyone who cares already knows, is an ARM chip running a slightly hacked Linux 2.6. If you install the Terminal Emulator app you can get a shell prompt: if you follow instructions on the xda-developers forum you can get root access too. I did: while I'm not sure it's essential to normal use, it does open up a couple of bits that you can't do on the regular phone such as wifi tethering and a task manager to kill apps you're not using when the thing starts feeling sluggish. (In theory this is unnecessary, in practice I'm not convinced either way but at the very least it's a great placebo effect)

But the bottom line is that it's fun to play with, which is probably the major thing, and seems to be quite low risk: at least, going by the comparative rarity of "I bricked my phone" posts on the xda-dev forum, most seem to have managed it successfully. You can even install an entire Debian system that lives alongside Android, though I still can't think of a better reason to do so than bragging rights.

Gnus over ssh#

Mon Jun 22 15:23:11 2009

Suppose you had a broadband account with a static IP address (or small network) and an NNTP service, and that you wanted to read news from places other than the broadband network. Suppose further that your newsreader of choice is Gnus.

The intersection of all these assumptions is that you'd probably be me. In which case you already have incoming ssh access to the machine(s) on the broadband and all you need do is add the emacs magic to connect back from wherever in The Cloud(sic) you are through your home net to your news service

(defun nntp-open-via-ssh-stream (buffer)
  "Open a nntp connection by ssh to coruskate then then nc to zen"
  (let ((command '("ssh" "me.homeip.example.com" "nc" "news.myisp.example.com" "119"))
	proc)
    (setq proc (apply 'start-process "nntpd" buffer command))
    (save-excursion
      (set-buffer buffer)
      (nntp-wait-for-string "^\r*20[01]")
      (beginning-of-line)
      (delete-region (point-min) (point))
      proc)))
(setq gnus-select-method '(nntp "news"
				(nntp-open-connection-function 
				 nntp-open-via-ssh-stream)))

I was a bit surprised there's aparently no canned function for this. I crafted this by cut and paste from nntp-open-telnet-stream, which seemed simpler than trying to bend it to my will by editing the variables therein

Zen Internet come highly recommended as usual