Turning the nftables#

Fri 02 Jun 2023 10:16:32 PM UTC

In the course of Liminix hacking it has become apparent that I need to understand the new Linux packet filtering ("firewall") system known as nftables

The introductory documentation for nftables is a textbook example of pattern 1 in Julia Evans Patterns in confusing explanations document. I have, nevertheless, read enough of it that I now think I understand what is going on, and am ready to attempt the challenge of describing

nftables without comparing to ip{tables,chains,fw}

We start with a picture:

This picture shows the flow of a network packet through the Linux kernel. Incoming packets are received from the driver on the far left and flow up to the aplication layer at the top, or rightwards to the be transmitted through the driver on the right. Locally generated packets start at the top and flow right.

The round-cornered rectangles depict hooks, which are the places where we can use nftables to intercept the flow and handle packets specially. For example:

• if we want to drop packets before they reach userspace (without affecting forwarding) we could do that in the "input" hook.

• if we want to do NAT - i.e. translate the network addresses embedded in packets from an internal 192.168.. (RFC 1918) network to a real internet address, we'd do that in the "postrouting" hook (and so that we get replies, we'd also do the opposite translation in the "prerouting" hook)

• if we're being DDoSed, maybe we want to drop packets in the "ingress" hook before they get any further.

The picture is actually part of the docs and I think it should be on the first page.

Chains and rules

A chain (more specifically, a "base chain") is registered with one of the hooks in the diagram, meaning that all the packets seen at that point will be sent to the chain. There may be multiple chains registered to the same hook: they get run in priority order (numerically lowest to highest), and packets accepted by an earlier chain are passed to the next one.

Each chain contains rules. A rule has a match - some criteria to decide which packets it applies to - and an action which says what should be done when the match succeeds.

A chain has a policy (accept or drop) which says what happens if a packet gets to the end of the chain without matching any rules.

You can also create chains which aren't registered with hooks, but are called by other chains that are. These are termed "regular chains" (as distinct from "base chains"). A rule with a jump action will execute all the rules in the chain that's jumped to, then resume processing the calling chain. A rule with a goto action will execute the new chain's rules in place of the rest of the current chain, and then the packet will be accepted or dropped as per the policy of the base chain.

[ Open question: the doc claims that a regular chain may also have a policy, but doesn't describe how/whether the policy applies when processing reaches the end of the called chain. I think this omission may be because it is incorrect in the first claim: a very sketchy reading of the source code suggests that you can't specify policy when creating a chain unless you also specify the hook. Also, it hurts my brain to think about it. ]

Chain types

A chain has a type, which is one of filter, nat or route.

• filter does as the name suggests: filters packets.

• nat is used for NAT - again, as the name suggests. It differs from filter in that only the first packet of a given flow hits this chain; subsequent packets bypass it.

• route allows changes to the content or metadata of the packet (e.g. setting the TTL, or packet mark/conntrack mark/priority) which can then be tested by policy-based routing (see ip-rule(8)) to send the packet somewhere non-usual. After the route chain runs, the kernel re-evaluates the packet routing decision - this doesn't happen for other chain types. route only works in the output hook.

Tables

Chains are contained in tables, which also contain sets, maps, flowtables, and stateful objects. The things in a table must all be of the same family, which is one of

• ip - IPv4 traffic

• ip6- IPv6 traffic

• inet - IPv4 and IPv6 traffic. Rules in an inet chain may match ipv4 or ipv6 or higher-level protocols: an ipv6 packet won't be tested against an ipv4 rule (or vice versa) but a rule for a layer 3 protocol (e.g. UDP) will be tried against both. (Some people [who?] claim this family is less useful than you might first think it would be and in practice you just end up writing separate but similar chains for ip and ip6)

• arp - note per the diagram that there is a disjoint set of hooks for ARP traffic, which allow only chains in arp tables

• bridge - similarly, another set of hooks for bridge traffic

• netdev - for chains attached to the ingress and egress hooks, which are tied to a single network interface and see all traffic on that interface. This hook/chain type gives great power but correspondingly great faff levels, because the packets are still pretty raw. For example, the ingress chain runs before fragmented datagrams have been reassembled, so you can't match e.g. UDP destination port as it might not be present in the first fragment.

There's a handy summary in the docs describing which chains work with which families and which tables.

What next?

I hope that makes sense. I hope it's correct :-). I haven't explained anything about the syntax or CLI tools because there are perfectly good docs for that already which you now have the background to understand.

Now I'm going to read the script I cargo-culted when I wanted to see if Liminix packet forwarding was working, and replace/update it to perform as an adequate and actually useful firewall

nftables or fn stable#

Tue 20 Jun 2023 08:11:36 PM UTC

Hadn't realised it had been so long since I last updated this, but progress has been slow lately.

liminix-rebuild seems to work nicely, and I have been dogfooding by using it to get the rotuer configuration into a shape that I could actually use it. So far: it runs PPPoE, gets IPv4 and IPv6 addresses, gets an IPv6 prefix using DHCP6 and advertises that prefix to devices on the LAN - so, albeit in an unpolished (verging on sketchy) fashion, it kinda sorta works. A simple firewall is currently in development, after having spent some time figuring out how nftables works.

Needed: (1) more polish; (2) better organisation. Once it works, lots of this stuff will need lifting into modules or services or both - or some other new yet-to-be-discovered abstraction - but the dependency there is on making it work first. I'm thinking to do an L2TP-based config as well as PPPoE (and maybe something with a wireguard VPN), which I think will expose more ordering/dependency requirements.

Housekeeping

Since NixOS 23.05 is now a thing, I'm also in the middle of making Liminix build/work with it. There's been a CI job running against unstable since I started, which has reduced the number of surprises, but still had a few things to work through:

• my disproportionate optimization to avoid using Libc in the initramfs no longer compiles - but 70k is no longer the minimum executable size anyway, so this makes the preinit go from about 5k to about 17k

• OpenSSL no longer builds, haven't yet worked out why. But this is not a catastrophe as nothing in the rotuer config should need it.

• using the NixOS Fennel packaging was causing it to try to build Python and all manner of other stuff. Fennel can be had from its upstream as a simple single Lua file, so in the circumstances we just use that.

We didn't start the fire(wall)#

Mon 26 Jun 2023 10:25:12 AM UTC

tl;dr:

• Liminix now works with Nixpkgs 23.05
• We almost route packets
• Docs have been tested and found wanting

First news for this week is that Liminix builds green in CI for all targets, and boots successfully on the GL-AR750. The only thing left to do on that score since last week was fix a bug in the preinit code that was not NULL-terminating an array correctly. It's presumed to boot also on the other hardware but I haven't tested it in prod yet.

The focus of development this week has been "CPE wifi router by any means possible", with the intention that we will extract modules and services from the running system once we have ... a running system. And I think we're nearly there.

• The IPv6 firewall is loosely based on the requirements in RFC 6092 and "works in principle" (I realised late last night that I was blocking DHCP client support on the WAN and therefore not getting a network allocated from the ISP: fixed this but it does suggest that the rules need review at a time I'm actually awake). What we don't have is a default route, because ... I don't know why not. Something I'm getting wrong with ip route add syntax or semantics, and the rather opaque error reporting is not helping. ip: RTNETLINK answers: Invalid argument. Quite.

• For IPv4 we have basic connectivity using NAT. It works but it's not fancy and it probably needs checking to make sure we're dropping bogons and generally being a good netizen.

To shorten the ntables development feedback loop, which often involves finding out that something doesn't work because it's missing from the kernel, I've added some code for building kernel modules without having to rebuild the kernel. This means we can add kernel modules as service dependencies and they will be loaded/unloaded as required by the service. It also means we can liminix-update to test new versions without having to reflash the kernel.

liminix-update could still be faster if it didn't have to reboot: resetting the board seems to take an age. I can see two avenues of investigation here:

• for changes to services, some mode of operation that would stop all running services, run the new config's activate command and then restart them.

• for changes that only add/remove packages, there is no need to restart anything, but to make the package accessible to the user (on their PATH) we do at least need to rewrite their profile. So, something that would do that.

The former of those would be more use to me currently: the only need I've had yet to install packages that aren't used by services is "oh, I'm missing tcpdump and strace".

....

The other area that needs some love is the new user/onboarding experience: the User Manual describes some things as they were in March, some things as they will be in The Future, and some things not at all. Going to give that some proper thought as soon as I have a router sufficiently functional that I can use it as I write.