> For 25 Gb/s Ethernet links, a commercial router is unlikely to be much cheaper or to consume much less than a standard PC.

EDIT: I linked to a switch by mistake.

This router has 12 x 10G SFP+ and 2 x 25G SFP28 ports and costs $595

It appears to be routing between 13 to 39Gbps depending on the rule complexity.

https://mikrotik.com/product/ccr2004_1g_12s_2xs#fndtn-specif...

This is a switch and switching is hardware accelerated. While you can theoretically route with this this, it is incapable of routing at gigabit speeds, let alone 10 gigabit speeds, once you need even a few rules.

Actually that switch has hardware accelerated routing as well with RouterOS 7 IIRC. Take a look at the test results, for large packets that router will still do over 10G throughput with 25 ip filter rules. With no queues or rules it'll do routing just as fast as switching.

Edit: Looks like he changed from the switch to a separate router but look at the change log for the latest R7 release

https://mikrotik.com/download/changelogs/development-release...

>!) added L3 HW support for all CRS3xx devices;

So you get L3 even for this monster. https://mikrotik.com/product/crs326_24s_2q_rm

Interesting. I had only looked at RouterOS 6. I have had trouble with mikrotik software before though. (I have a CSS 610 and a lot of its core features were completely broken before)

> After my previous disappointment with the MikroTik CCR2004, I decided to try a custom PC build.

The article's author considered that exact router and rejected it.

He goes into greater detail here: https://michael.stapelberg.ch/posts/2021-05-28-configured-an...

That‘s a switch, not a router. Much simpler hardware.

Here‘s a MikroTik Router with 28 Gbit/a throughput: https://mikrotik.com/product/CCR1036-8G-2SplusEM

$1300 and up to 73W

That's a bit of an odd example to choose.

This[0] has a $600 MSRP, with twelve 10Gbps ports and two 25Gbps ports. It's ~32W before you add the SFP+ modules, and upwards of 50W if you have all modules populated.

In the benchmarks they list, this can provide somewhere on the order of 13Gbps to 40Gbps of routing, depending on exactly what you're doing. (Smaller packets will lower these numbers, but if you care about bandwidth you're unlikely to be worried about smaller packets... at that point, you probably care more about kpps.)

[0]: https://mikrotik.com/product/ccr2004_1g_12s_2xs

the benchmark you even link shows absolutely abysmal performance curves, dropping down into even half a gig perf for small packets single stream.

Calling any benchmark "abysmal" requires a point of comparison, and you have provided none... so it doesn't really encourage good discussion. What's the point of your comment? It seems to just be a way of insulting someone else's product.

Which comparably priced router are you thinking of that has a "non-abysmal" performance curve for small packet bandwidth?

Fair. It was the first router I found on their site with listed throughout >= 25 Gbit/s.

SFP requires less energy than RJ45, only 0.7W/port. The MikroTik 4-port CRS305-1G-4S+IN draws 10-18W.

That's not entirely true. SFP power consumption depends on the type of SFP. A 10Gbps DWDM SFP+ might draw 1.8W of power. The why for this is actually quite interesting: the lasers used in DWDM SFPs have much more stringent requirements for temperature stability to ensure the light emitted doesn't drift out of spec. In order to achieve that temperature stability, they use a built in peltier to pump heat away from the laser and control the temperature. They're quite the marvel of modern engineering!

It would have been fun to be a fly on the wall at Cisco when they first realized people were jamming devices with heaters in SPF sockets.

Cisco didn't "realize" this was happening, they specified, had manufactured, and sold the very SFPs the GP post is describing. SFP slots in routers are designed with this thermal load in mind.

Which is also why plugging a large amount of DWDM optics into a datacenter switch is a bad idea. Datacenter switches are _not_ designed with this in mind. You run into risks of both overheating the switch as well as overloading the PSUs. A small number of high-power optics ain't gonna break the switch though.

And: 10Gbase-T SFPs have horrendous power consumption, even worse than DWDM SFPs. At these speeds, the signal over copper is mostly mush, and the PHY contains a none-too-trivial analog & digital signal processor. Which, again, is where limitations for 10Gbase-T SFP usage come from. If at all possible, avoid this shit — there's absolutely no reason to have 10Gbase-T inside a rack, for example. Just use DAC cables or SR optics.

I'm being a bit glib, but from the point of view of someone who used SFP optics once in a while, it seemed like the whole "standard" was a bit of a mess.

If it was known that vendors would try to push their thermals past what the device cage would support, why not address that better?

The general impression I was given is that thermal loads were one of the primary reasons given for vendor-locking pluggable optics, and that the original SFP spec didn't even address it.

Except no one is using DWDM for homelabs, as it's usually used for longer distances. The peltier cooler is definitely cool, though.

True. On general principle I would recommend folks invest in single mode rather than multimode fiber for permanent installs, as the price delta on SFPs is low enough these days. Multimode is a complete pain as it needs to be upgraded for higher speeds every decade. If it's just a couple of patch cables in a home lab, it doesn't matter, but if it's run through a wall...

It's also a real laser versus a many sfp+ gbics being LED light source now

> MikroTik 4-port CRS305-1G-4S+IN

I have two of these at home, they're neat little bits of kit.

They're fanless and therefore silent, which is fine until you realise you want to do 10GbE over existing copper cables with something like Mikrotik's S+RJ10 adapter. Then the temps start to rise...

So, I've decided to go completely to fibre, even if it means opening up the walls of the house. Just bought a job lot of used ConnectX-3 cards off ebay.

> ConnectX-3 cards off ebay.

Those cards are so good. I got a few after seeing them rated in r/homelab and haven’t looked back. The Synology took one and it was about 1/10th the price the official Synology one was.

> Those cards are so good

Agreed. You can find them on eBay for ~$40 a pop, and they work flawlessly on Linux and Windows. I've made a few purchases from the seller "bitsquad" (ships from Kentucky), and he always seems to have Mellanox/Chelsio in stock.

Another good tip is to use Direct Attach Cables (DAC), which have a much lower power consumption and save you the cost of 2 fiber optics as well. This only works on short distances of course, but between the MikroTik and a few devices that might very well be the case.

How do you obtain custom lengths of fibre cables that work with ConnectX-3?

Use fs.com, their generic cables work with MikroTik no issue and they do custom orders.

The ConnectX-3 cards are not vendor locked in any way. (And even if they were, that'd only affect the SFP, not the cable.)