This week I successfully flashed NixWRT to my GL-MT300A, such that it
runs whenever I turn the device on. And then I found it slowly fills
up all RAM and the process table over about half a day then stops
working.
But we'll get to that in a minute. Let's talk about the good bits
first.
Graven image
Given we have a kernel and a root filesystem, how exactly should we
mash them up into a flashable image such that (i) uboot will find and
run the kernel, (ii) the kernel will find its filesystem? Let's look
at the dmesg output from booting OpenWRT -
[ 0.620000] m25p80 spi32766.0: w25q128 (16384 Kbytes)
[ 0.630000] 5 ofpart partitions found on MTD device spi32766.0
[ 0.630000] Creating 5 MTD partitions on "spi32766.0":
[ 0.640000] 0x000000000000-0x000000030000 : "u-boot"
[ 0.650000] 0x000000030000-0x000000040000 : "u-boot-env"
[ 0.650000] 0x000000040000-0x000000050000 : "factory"
[ 0.660000] 0x000000050000-0x000000fd0000 : "firmware"
[ 0.780000] 2 uimage-fw partitions found on MTD device firmware
[ 0.790000] 0x000000050000-0x000000174720 : "kernel"
[ 0.790000] 0x000000174720-0x000000fd0000 : "rootfs"
[ 0.800000] mtd: device 5 (rootfs) set to be root filesystem
[ 0.810000] 1 squashfs-split partitions found on MTD device rootfs
[ 0.810000] 0x000000890000-0x000000fd0000 : "rootfs_data"
[ 0.820000] 0x000000ff0000-0x000001000000 : "art"
It has five "ofpart" partitions. I'm guessing the "of" stands for
"open firmware" and indeed if we look at the DTS file (remember if you
will from blog entries passim that the device tree is a representation
of "what you'd get from open firmware if the hardware had open
firmware") we can see five partitions defined there.
It has two "uimage-fw" partitions which further subdivide the
firmware partition. Unlike the ofpart partitions these are not
actually defined anywhere: they are the result of kernel code (in
drivers/mtd/mtdsplit/mtdsplit{,_uimage}.c) which looks for
partitions which start with a uimage, parses the image length, and
then looks for a filesystem signature of some kind on the next erase
block boundary. (This is very convenient magic not least because it
means we don't have to update some partition table each time our
kernel size changes, but it still makes me uneasy; I have a very low
threshold for magic)
Hypothesis (subsequently proven): if our firmware file consists of a
kernel wrapped in a uimage, plus padding to the next erase block
boundary, plus the filesystem image, Linux will report an MTD
uimage-fw partition that starts where the filesystem starts. We can
copy this combined image into flash at offset 0x000000050000 to
overwrite the existing kernel/root fs while leaving the rest of the
flash undisturbed.
(To get the erase block size, we could (a) guess it's probably 128k, or
(b) if we were more prudent, check in in /proc/mtd )
Short version: that half of the puzzle is solved by creative use of dd.
Flash override
So the general principle for flashing from U-Boot once you have a
suitable image, is (1) download it into RAM somewhere, (2) erase an
appropriate section of flash, (2.1) hope the power doesn't fail before
you finish step 3, (3) copy the image from RAM into flash, (4) reboot
and see whether you've bricked the device. Although unless you've
done something badly wrong (like overwrite uboot itself or the ART
partition) then it doesn't matter too much if the image you've
uploaded doesn't actually work because you can just go back to the
u-boot prompt and try again. There's an explanation of how to do
this on the Yun
which I had previously successfully followed, converting that to
another board is just a matter of working out whereabouts in RAM the
flash chip is mapped.
And the simplest way of doing this is to look at what uboot does by default
when the board is powered on. If we run printenv we see i.a.
bootcmd=bootm 0xbc050000
With 99.8% certainty, we think that this is a jump to offset 0x50000
(remember, this is the offset of the "firmware" partition) of a flash
chip that starts at 0xbc000000, and this is probably all we need. So:
on the build machine
and then offer up a silent prayer because 99.8% is still less than 100%.
Punch the air shortly thereafter :-)
Flash I love you but we only have 14 hours
As alluded to above, there is something weird going on in userland
that makes it presently less than useful: the ntpd and syslogd
processes (both actually BusyBox applets) don't write pid files when
they start up, causing monit to decide they have failed to start and
spawn another. One of each of them added every 30 seconds soon leads
to a poorly computer.
No idea why, yet. I'd run strace but it doesn't want to build (maybe
a MIPS thing, maybe a musl thing). Hopefully next week...
Completely random aside
And I mean completely random. Is it just me or does anyone else
find that the taillight cluster on the new Prius reminds them of Ming
the Merciless?
See subject. Most of the work this last week has been moving things
around in the hope of making it possible to support more than one
device, and then merging the gl-mt300a branch into master (nitpick:
for reasons I can't remember and are unlikely to be convincing, the
primary development branch is actually called nixwrt not master.
Probably something to do with git filter-branch)
This breaks the Yun code that was previously there, because "possible"
and "actually implemented" are two different things. But I am not
using it, I am pretty certain nobody else is either, and at least now
I can see how to fix it.
I have added a targetBoard argument which will soon allow choice of
mt300a, yun or malta (that last one for qemu), so the build command is
presently:
I have added a `swconfig` invocation to the monit config so that
networking comes up. I've done it in a totally hacky way until I
decide how to represent the switch in configuration, but that might
involve learning how the damn thing works.
That's about all for now, save to say that yesterday I went to the
NixOS London
meetup which in
the event did not involve an install party (everyone present had
installed it already) but did involve some interesting conversations,
learning something about overlays and giving a quick demo of NixWRT so
far. Albeit that I was demonstrating by ssh back to my hardware at
home : no actual hardware at the venue that could be waved around or
kicked. I had intended to do something a bit more visual but thought
better of it when I realised how much of my home kit I'd have to
unplug. Next time, definitely.
[ meta: I wanted to call this one "sudo make me a LAN switch" but now the
MRAs have ruined that phrase for everyone ]
I got networking working on the GL-MT300A. This entailed:
Patching the device tree definition.
This is slightly cargo-culted (I Read It On The Internet), but by
removing the pinctrl-0 entry from the ðernet stanza I managed
to change the bootup mesages from saying
[ 1.873672] rt2880-pinmux pinctrl: could not request pin 40 (io40) from group
ephy on device rt2880-pinmux
[ 1.883620] mtk_soc_eth 10100000.ethernet: Error applying setting, reverse things back
[ 1.891724] mtk_soc_eth: probe of 10100000.ethernet failed with error -22
to saying
<6>[ 2.586201] mtk_soc_eth 10100000.ethernet eth0 (uninitialized): port 1 link up (100Mbps/Full duplex)
<6>[ 2.595753] mtk_soc_eth 10100000.ethernet: loaded mt7620 driver
<6>[ 2.602600] mtk_soc_eth 10100000.ethernet eth0: mediatek frame engine at 0xb0100000, irq 5
which felt a lot like progress but did not result in actual connectivity.
Building swconfig
After trying the obvious culprits (firewall rules? weird routing?) for
why my board was not seeing the network - indeed, not even able to
ping its own IP address - I studied the dmesg output a bit more
closely, and noticing the line
<6>[ 2.581845] gsw: setting port4 to ephy mode
I took a wild-ass guess that given I knew the the device contains
some kind of network switch, maybe the switch doesn't come up in a
useful state. So we needed a tool of some kind to reconfigure it and
apparently the appropriate tool is
swconfig
(If you followed that link and struggled to understand what it was
talking about, be assured that it means nothing to me either. Ure not
alone)
Building swconfig is easier when you start with a fork for
Debian instead of the original
OpenWRT package: I simply made my kernel derivation install the header
files (first time I have written a multi-output derivation, but turned
out that in this case it was a one-line change), wrote a derivation with
libnl as a dependency, and
created an 80MB filesystem image.
PRO TIP: don't override phases in a Nixpkgs derivation unless you
understand what is done by all the phases you didn't include. In
this case, not running the fixup phase meant that
nothing ran the "shrink rpath" magic which removes unneeded
compile-time dependencies from the runtime dependency list. Image
size go boom.
Running swconfig
I spent some time trying to figure out what I was doing with vlans and port configuration and worrying that when it said link: ?unknown-type? against each port that would mean I had to tell it somehow what kind of link type to use. Then eventually I hit on
swconfig dev switch0 set enable_vlan 0
swconfig dev switch0 set apply
and as if by magic (= "insufficiently advanced understanding of
technology") it all started working. When I eventually want it to
function as a switch, obviously I will need to revisit this. But that
is Milestone 1
and this is Milestone 0 - sufficient unto the day etc etc.
(Once I got warmed up, at least. I returned from my holiday to find
that the entire local network had stopped working because after
mucking around with U-boot on the device while I was away, I'd
inadvertently let the default openwrt installation on the MT300A start
, and it was running a DHCP server. Shouldn't have put it on the LAN,
I suppose. One round of reboots later ... but apart from that it has
been a productive week)
Let's start by spoiling the ending, so you don't have to read the rest of this
post: said MT300A now boots to user space and runs init (and monit). I hope
that all that remains is to get the Ethernet working and to build a flashable
image.
On our way to that destination, we ... basically, this is another
thrilling instalment of "don't trust the bootloader"
This board builds with device tree, but its u-boot has no way to
provide a device tree blob at start up, so what we have to do instead
is bodge the device tree into the kernel itself.
The actual mechanics of glomming a device tree binary blob onto the kernel image
are fairly straightforward if you have an openwrt build to crib from:
generate the ELF vmlinux as usual, convert it to a raw binary, compile the DTS
file (which involves preprocessing it with cpp because - I don't know why -
it contains two different kinds of include directive and the dtc tool only
understands one of them) and then use some magic patch-dtb tool to stick
them together.
I previously described device tree as "let's pretend that the hardware
description was provided to us by open firmware." and insofar as this
means the hardware is described by a data file instead of by the
effect of running imperative C code then it's definitely good. But
when that data file is provided by the kernel source [*] and attached
to the kernel image, instead of being passed in by the bootloader as
an input to the kernel, things get weird. For example, the default
config for the MT300A is that the kernel command line is coming from
DT - effectively this means that the kernel provides its own command
line. And you may ask yourself: where does that highway go to? It
would make sense if the command line had been provided by the user to
the bootloader which then merged it into the DT, but in this scenario
... not so much.
[*] splitting hairs here: the particular DTS we're using comes from
LEDE and not from the mainline kernel. But that doesn't invalidate
my point, which is that it doesn't come from the bootloader.
At this point, and skating over a minor digression where I had accidentally
built a kernel that thought it was little endian but using a big endian
compiler (tl;dr - didn't work), I had a kernel that booted most of the way to
mounting root but then failed.
The reasons it failed were approximately legion in number, or at least that's
how it felt. In the order I discovered them :
I. The kernel was configured to get its command line - see above - from DT not
the bootloader, so was ignoring all my phram options because the hardcoded
command line overrode them.
II. For reasons that no doubt made perfect sense to the people who
were there are the time, the u-boot build on this board doesn't
support bootargs anyway. So the kernel was ignoring all my phram
options because it wasn't even seeing
them
. Remember, kids - the U in "u-boot" stands for #undef . (I am
assuming the sources there correspond with the binary that shipped on
my device, to be quite honest I haven't checked this properly for myself).
III. another bit of hardcoded magic ... there is a kernel config option in
LEDE, again enabled by default, that looks through the MTD partition table and
if it finds a partition named rootfs, sets it up as the root filesystem.
This overrides any root= option given on the command line. So the openwrt
rootfs on the actual flash was being used instead of the custom root fs in
phram - and failing to work not least because it's a jffs filesystem and I'd set rootfstype=squashfs.
IV. Whereas the Yun wanted the address of the filesystem image as an
offset from 0x80000000, the MT300A wants it offset from 0. I haven't
got the MIPS memory map entirely straight in my head but i think I
would be using the right words if I said it wants a kuseg memory
address not a kseg0 address. Or perhaps it wants a physical address
not a kseg0 address. (The same physical RAM is mapped in both places
but the cache behaviour is different). Whichever is the right
explanation I don't know, but when I fixed that, bingo, it boots!
I don't know if the Yun would work with kuseg addresses too, but next time I plug it
back in I'll try it.
Also this week, significantly cleaned up how we provide config
options to the nixwrt kernel
derivation
- now we read the appropriate defconfig file(s) into an attrset then
pass that attrset into overrideConfig. overrideConfig is a
function that accepts an attrset of default config options and is
expected to return a customised set. It's a lot prettier than the
rather awful mess of echo and grep -v it replaces, but it's not
100% perfect because it doesn't know the type of each config option
value - if you have a string value you need to quote that string
yourself (tip: use builtins.toJSON) but it will do for now.
Probably I should extract it into a function so that I could use the
same code to configure Busybox.
That's about it for this week.
With luck and a following wind I'll be at the NixOS 18.03 install
party
at Codenode - I hope to bring along some hardware - so if you're
interested and in London, sign up and come along.
If you'd like to get NixWRT updates more than once a week, I
encourage you to follow me on Mastodon
My GL-MT300A arrived just as I was about to go on holiday. This is how far I've got -
Serial console
These things are, if not actually made for DIY purposes, at least
very tolerant to such uses. Take it out of its case and you find
three standard 0.1" header pins on the PCB labelled "TX", "RX", "GND"
- connect each of them to something that speaks TTL serial (I used a
Raspberry Pi) and set the baud rate to 115200. Worked first time.
"The U is for Uninitialized"-Boot
I commented previously about the differences one may encounter
between two devices both of which run the allegedly "Universal" U-Boot
boot loader. This time I couldn't work out why my tftp downloads were
loading into memory at offset 0 instead of, say, 0x811f8000. Until I
realised that (i) it no longer sufficed to say
and (ii) on this device, double quotes around the value of a setenv are no longer special, so
setenv bootn "foo;bar"
will set the value of bootn to "foo and then attempt to run the command bar". Which typically doesn't work all that well.
Hello darkness my old friend
Having made the relevant changes I was able to get the following output:
## Booting image at 81000000 ...
Image Name: Linux-4.9.76
Image Type: MIPS Linux Kernel Image (lzma compressed)
Data Size: 1705466 Bytes = 1.6 MB
Load Address: 80001000
Entry Point: 803fa9c0
Verifying Checksum ... OK
Uncompressing Kernel Image ... OK
No initrd
## Transferring control to Linux (at address 803fa9c0) ...
## Giving linux memsize in MB, 128
Starting kernel ...
followed by indefinite but emphatic silence, and various bouts of
fiddling with CONFIG_EARLY_PRINTK and stuff have not yet persuaded
it to loosen up. Currently I am running LEDE in a Docker container to
see what it does, and diffing its .config with mine. This has
shown up a couple of things that I've now added to my configuration, but I am
only going to get one shot at running it before I go home, because at
70 miles distant from the hardware I can't reach across and power
cycle it.
This week I successfully flashed NixWRT to my GL-MT300A, such that it
runs whenever I turn the device on. And then I found it slowly fills
up all RAM and the process table over about half a day then stops
working.
These things are, if not actually made for DIY purposes, at least
very tolerant to such uses. Take it out of its case and you find
three standard 0.1" header pins on the PCB labelled "TX", "RX", "GND"
- connect each of them to something that speaks TTL serial (I used a
Raspberry Pi) and set the baud rate to 115200. Worked first time.