I can’t recommend the CM4 over anything with an actual switch built into it with network fabric providing CPU-free packet switching and forwarding. You also honestly don’t want something as important as your actual internet gateway to be some hacky thing that might or might not be supported in a few months or years, unless it's just for experimentation in which case knock yourself out.

Unfortunately I don't think there are any implementations of CPU-free QoS, which you really want at the network boundary.

That really depends on the used SOC, or how well its hardware features are supported under Linux, by non-BLOBs, preferrably.

Anything MARVELL should mostly do, usually. But that's not the cheapest segment anymore, so the people are hurting themselves by buying the common plastic trash. Or use DD-WRT instead of OpenWRT, because they give a shit about binary-blobs and integrate that however they see fit, because they prefer to be able to use everything the hardware supports, instead of having an almost brick running open-source.

I don't think any devices supported by OpenWRT, DD-WRT, VyOS, pfSense, etc support hardware-accelerated routing with QoS. If you know of one, let me know!

It is not always clear, but enabling QoS usually disables hardware offloading, for example with Ubiquiti devices[1] "Traffic to which QoS policies have been applied cannot also take advantage of the EdgeRouter's Offloading feature". The same is true for open source even with binary blobs. The chips simply can't do it.

Using QoS and implicitly disabling hardware offloading significantly drops the throughput as it becomes CPU bound. "Without offloading enabled, IPv4 traffic will be routed via the CPU and will be limited to around 300Mbps on the EdgeRouter Lite (ERLite-3). With offloading enabled, the throughput will be about 950Mbps." [1]

[1] https://help.ui.com/hc/en-us/articles/115006567467

[1] https://macchiatobin.net/

[2] https://en.wikipedia.org/wiki/Turris_Omnia

[3] https://docs.turris.cz/hw/omnia/omnia/

These two and similar were what I had in mind.

Though I don't have them. But considered them a few years ago. But I've chosen another path, of really dark art ;-> (Don't ask)

edit:

Oh, and

[4] https://espressobin.net/

The smaller, and more affordable option from the makers of [1]

But this is all from three, or even four years ago?

I don't even know if that Turris Omnia thing is EOL meanwhile?

They wanted to make something even more modular.

Anyways, all MARVELL based and thus IIRC with working offloading under Linux, though not mainlined at the time.

[5] https://en.wikipedia.org/wiki/Marvell_Technology,_Inc.

edit: Funny, Amazon.com says they have 9 Turris in stock

[6] https://www.amazon.com/Turris-hi-Performance-printserver-Vir...

While amazon.de has 19.

So they should be available. But about 400, instead of maybe 40 to 200 for some cheap plastic trash which melts or resets if you look at it the wrong way.

One downside to the Raspberry Pi (CM)4 is that it doesn't have hardware crypto. So it'd make a poor VPN machine. But otherwise I think it'd be fine. Some people are happy just using a regular Pi 4 connected to a VLAN-capable switch or with a USB adapter for the second NIC.

I recently bought a GL.iNet Brume. It's a nice tiny ARM machine I use as a wired-only router. /proc/cpuinfo confirms it supports hardware crypto...

    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid

...although I haven't set up a VPN on it yet. Downsides: the WAN and LAN0/LAN1 ports are all the same switch with only one network interface, so it'd be impossible to achieve >500Mbps full-duplex. (It's never come up for me; my Internet connection's too slow.) It's also a little pricy for what it is.

I might have tried the FriendlyELEC NanoPI R4S instead, which actually has two separate NICs. But if I'd ordered that I'd still be waiting for it...

One thing I don't care about: hardware NAT/routing offload. I think that's often unsupported by OpenWRT, and it's incompatible with SQM.

Concerning HW NAT/Offload support: the DSA support is (iiuc) supposed to be the framework, through which the offload functionalities will be implemented. There is a pull-request for DSA even for architectures as far back as ath79.

I haven't read into the specifics yet, but it sounds like nft-tables configuration will be translated to a eBPF program, that will then hook into network flows, and tell the switching HW/NPE over some side channel which accelerations to use and shortcuts to take.

But do I really want to trust a "Black Box Network Processing Engine" bolted onto my CPU, that can route packets entirely outside the realm of control of the main CPU?

I always wonder about intentional or unintentional "in-band debug features", when people mention, that they run their home-lan on the same switch as the WAN-Uplink. I mean, is there really a HW vendor, that we trust to get the switching logic 100% correct/secure/performant?

My understanding from the kernel docs [1] is that DSA is just for switching, not routing. Do you have a citation that suggests otherwise?

My GL.iNet Brume supports DSA on its switch with OpenWRT 21.02, as does my older Netgear WNDR3800.

I'm not sure there's any hardware/software combination out there that supports offloading NAT/routing with proper QoS. So I'm puzzled when people talk about the importance of offloading or suggest using machines with very low-power CPUs. The Brume is Dual-Core ARM Cortex-A53 @1.0GHz, which gets the job done for me, but I wouldn't want to go any slower than that.

[1] https://www.kernel.org/doc/Documentation/networking/dsa/dsa....

The link you posted mentions "setting up offloads", and unifying the functionality of DSA and switchdev.

To me, DSA (as a "convenient" means to access/name/treat individual switch ports) is a prerequisite to enable usage of whatever Switch-Chip-Level-Offloads the switch chip supports. Without DSA, vendors had to use nasty hacks to make use of e.g. HWNAT, which is why there never was mainline support for these acceleration features.

HWNAT is a routing function, and many/most? switches of the last decade support that. But, OK, true, most of them will probably not be able to HWNAT "with proper QoS". But newer devices are not "just" a switch-chip, they contain a "Network processing engine", which is not a "fixed-function" device, but rather a "programmable" accelerator, that you could perhaps even "teach" how to do PPP, Encrption/VPN wrapping, and/or "offloading NAT/routing with proper QoS".

And now that DSA and eBPF are in mainline, using eBPF to generate a program/set_of_rules for the Ethernet acceleration device from your nftable-rules seems like the natural next step for me.

Or where do you see my reasoning go off the rails?

To paraphrase: you're saying that DSA doesn't support offloads, and most HW NATs don't do QoS, but newer hardware is capable of it, and you expect mainline Linux to make full use of it in the future with DSA.

No, I don't know of any problem in your reasoning. I'll look forward to that happening! I wasn't aware newer hardware was so flexible.

Yes, nearly: DSA currently only supports a few basic offloads, and all?/most of those only work on some of the supported DSA targets. I think, MT76xx devices are the most "advanced" in that respect right now. But nothing "advanced" like even pppoe-offloading (yet).

I just hope, "flexible" doesn't come with a large "Difficult-to-audit-caveat" side-dish. NPEs "governing" traffic could potentially substantially reduce the trust the user can have towards the platform.

Not sure if it supports all the features you need, anyway you might find interesting the PCEngines APU2 boards.

https://pcengines.ch/apu2.htm

Anyone has experiences with the complete build from https://teklager.se/en/products/routers/

They look very nice (although unavailable till January)

Those are neat looking machines. I saw them and iirc quickly ruled them out because they're not available now.

I found a passively cooled x86 box that I run openwrt on. It's got 6 Intel gigabit NICs and I use Proxmox to virtualize guests. OpenWrt is one of those guests, as is Home Assistant which controls various home automation things. I posted about it on Reddit a while ago: https://www.reddit.com/r/homelab/comments/hzvfih/new_router_...

At the time it had much better hardware at a similar price to more popular brands like Protectli and Qotom. Not sure what the scene is currently, but there a bunch of similar offerings on AliExpress.

Why did you choose x86 openwrt over opnsense? I chose opnsense because it never occurred to me to run openwrt on not-a-consumer-router and now I'm wondering if I screwed up.

Like you, I run a single x86 box with a handful of guests for random stuff - including Home Assistant!

openwrt is/was better for fq_codel and cake which were linux things. There may also be other packages that you may find on one or the other. I make extensive use of vpn-policy-routing on openwrt which does selective routing over a few different vpn interfaces (wg, openconnect, etc.). Not sure if something like this exists in bsd land.

I did try opnsense, but found that I wanted wireguard more than any of the features it offered. At the time wireguard wasn't a built-in option, so went back to OpenWrt which I was already familiar with anyway from running it on Linksys routers.

I had a similar experience and ended up doing the same. OPNsense does include a userspace implementation of Wireguard, but is obviously inferior to what Linux already has. It ended up being that, how poorly the initial kernel space implementation was handled for FreeBSD and that I wanted Cake for QoS, which made me move back to OpenWRT.

I do want to ask though, has there been any downsides to using OpenWRT on x86?

Late reply but -

I haven't really run into notable downsides. Installation was somewhat of a pain since I had to resize the partition (all of the prebuilt images assume small hardware), and all of the default packages are as minimal as possible - like busybox for userspace/shell. Upgrading on X86 is I guess possible (I haven't tried yet), but most of OpenWRT's upgrading instructions/support are geared towards reflashing firmware, not updating packages on ext4.

But, there are a bunch of available packages - coreutils, zsh/bash, etc., that can be installed as desired. In general OpenWRT is pretty set-it/forget-it for me.

Makes sense! I'm using wireguard on opnsense (but it's a newish feature) and so far it's been a great experience.

If you want something fast and inexpensive than wait for OpenWRT build for Redmi AX6 / AX3600 WIFI routers. For now only manual build is available and some have to reboot from time to time due to memory leak.

https://forum.openwrt.org/t/adding-openwrt-support-for-xiaom...

Probably depends on how complex your setup will be. We are running a normal 1GB Pi4 with OpenWRT (trunk until now) for quite a while on a 1gbit down/500mbit up fibre uplink with SQM against bufferbloat. Unifi switches handle all VLAN related work, fibre uplink is on VLAN 7 mandated by the carrier, so PPPoE works with only one LAN port on the Pi4. Wifi is also unifi hardware.

Thanks to full duplex, clients are able to use the fibre uplink without slowdown. VPN is managed by a separate system, though we had some successful experiments with WireGuard. Both a colleague and I are running the exact same setup at home (with fibre uplink and VDSL).

It's a very capable and power efficient setup. I'd definitely go with a CM4+dual NIC board today.

If you're looking for a router that does NAT you're not going to beat a purpose built hardware board in terms of perf per dollar, especially only latency and jitter of the NAT.

If you're looking for a network connected server that happens to do a lot of packet processing you can go the rpi route but I'd still front the internet and rest of the home internet off a small cheap hardware box like the Hex.

I moved from an edgerouter4[1] to openwrt on a protectli fw4b. On my home gigabit fiber (centurylink) link, it has been great so far (performance, throughput, etc)!

[1]: Ubiquiti has had packet processing issues in the 2.x firmware series for what seems like forever, and lack of updates lately has prompted me to move to something that is arguably better anyway.

Both built-in interfaces of the RPI4 don't support MTU over 1500. That rules out many advanced usage scenarios, I think.

They can be modified, though not necessarily in a typical way. Also, CM4 implementations vary, and most interfaces on 3rd party boards can have MTU set as normal.

Hi, thanks for that correction. Do you have a keyword or 2 to google for "They can be modified, though not necessarily in a typical way." Or is that only or the CM, and not the normal RPI4?

I only found some quick&dirty patch, that seems to make MTU up to 2000 functional. But since the patch aims for 9018, and only gets 2000, that does not seem like the "right" solution.

You can do it if you compile in a kernel patch on the regular Pi 4, I believe: https://www.jeffgeerling.com/blog/2020/setting-9000-mtu-jumb...