Argumentum ad arborem fabrica#

Wed, 25 Jul 2018 22:54:16 +0000

What I'd like you to take away from this post title is that I speak about as much Latin as I do German.

What I'd like you to take away from the post body is (i) that I have a solution for the problem it describes, and (ii) that it required a tonne more of reading code and adding debugging statements and experimenting than I think it reasonably ought to have done, so look upon my words ye mighty and despair. The picture is a Google search result for "flattened tree", if you were wondering.

So, as I said previously we have (now, "had") a problem with kexec and specifying the command line arguments to the kernel: on the one hand we want to ignore any arguments that the bootloader provides, because generally they're probably wrong, but on the other we want to pay attention to the command line when booted by kexec, because the appropriate parameters for booting from flash are not also appropriate when booting from RAM. I'm going to skip over the voyage of discovery here because it's almost as tedious to relate as it was to, er, discover. So here are the highlights:

Kexec on MIPS (for ELF) provides two ways to supply the kernel command line.

The first option is that you add a segment which starts with the magic string "kexec " to the list of segments that you call kexec_load with, and then the pre-reboot kernel kexec code (machine_kexec_init_argv) iterates through the segments, finds the one with the right magic prefix, and parses it into kexec_argv[]. Then after the reboot, code in relocate_kernel.S loads the argument vector into register a1 before it calls into the new kernel. head.S in the new kernel then copies the pointer into fw_arg1, and then some board-specific code is responsible for what happens next. For the ralink case, this is prom_init_cmdline in ralink/prom.c which copies the argument vector back into a single string arcs_cmdline. After that, the next point of interest is in kernel/setup.c which tests a complex combination of kernel config options to decide which of arcs_cmdline, boot_command_line and builtin_cmdline (gotta love that consistent use of abbreviations) are used and in what combination to form the command line that the kernel will actually see.

There is a comment in the kexec source to say that this only works on an Octeon. Now that I trace the entire execution path I no longer understand why it only works on Octeon, but I will note that it didn't work for me. And, incidentally, wouldn't solve the problem if it did as we can't identify whether the command line came from kexec or u-boot. Anyway, taking inspiration from the said comment that this is "legacy", I decided to go with the second way.

The second way is to pass a DTB (a compiled device tree) from kexec, and embed a command line in there. There's a branch of the tree called chosen and within that is a leaf called bootargs, and that's where you find the command line. As a string, not an array of strings, please note.

In stock Linux there are two defined ways tell your kernel where its DTB is (in addition to anything your bootloader might do, if your bootloader is an accomplice rather than an adversary). The first option is to include it in the kernel as a special ELF section, or the second is to append it (using cat or similar) to a raw kernel image. It should be noted that the first approach only works if your kernel image is ELF (ours isn't) and the second - aside from being somewhat brittle if you ever boot a kernel where you forgot to concatenate the DTB - only works if your "raw image" is a zImage (ours isn't). So it's probably not at all surprising that OpenWRT have added a third way: in kernel/head.S they've added 16kB of zeroes preceded by the magic string "OWRTDTB:", then provided a utility called patch-dtb to run at kernel build time, that looks through the kernel image for this string and patches a provided DTB into place. This location is labelled __image_dtb, and for ralink boards there's some code in plat_mem_setup to call __dt_setup_arch on it.

(You will observe, if you're following all this, that this code is unconditional, so the third option is not so much an option as an override)

__dt_setup_arch calls early_init_dt_scan which calls early_init_dt_scan_nodes which calls early_init_dt_scan_chosen to populate boot_command_line.

After that, we're back to kernel/setup.c and the same complex combination of kernel config options we already saw, to decide which of arcs_cmdline, boot_command_line and builtin_cmdline are used - except that this time the answer we want is boot_command_line not arcs_cmdline.

So: how do we get kexec to inject a different DTB (which will in practice be a very similar DTB except for the /chosen/bootargs node) into this sequence? Three things.

First, the userland side . This turns out to be pretty simple if you have the ELF code to crib from - we read or create the DTB in RAM, and then add it as a segment to the segment list that kexec_load is called with.

Second, the pre-reboot kernel side. There is existing support in the MIPS "generic" kernel for finding this segment by stepping through the list looking for an fdt header . We're not running the generic kernel - we're running the ralink kernel - so we moved that code into mips/kernel/machine_kexec.c and made it conditional on CONFIG_USE_OF. The effect of this code, if it finds a DTB, is to put its address into kexec_args[1] and set kexec_args[0] to -2.

Third, after reboot the kexec_args[1] ends up in register a1 and then gets copied to fw_passed_dtb. All that remains after that is to change the code that hardcodes using image_dtb into code that defaults to image_dtb if we didn't get a DTB some other way - voici - and we're basically good to go. One yak successfully popped off the shaving stack.