If you have enough USB ports, you should be able to get away with using DisplayLink dongles for the video part. DisplayLink doesn't require any additional video processing hardware (unlike DisplayPort) or even USB-C ports, you can run the old DisplayLink standard over plain old USB 2! Not great for playing video, but a terminal app should run just fine.

I think the current driver restricts you to 4 displays but I'm not entirely sure why (you can do 6 on Windows so I think it must be a driver issue).

Of course, with your RPi idea, you could connect one computer to dozens or maybe hundreds of serial ports, but if all you want is five more screens, USB should do fine for your needs.

DisplayLink creates virtual displays on CPU and stream compressed feed over USB. That takes non-negligible amount of CPU time and RAM bandwidth, and it makes sense to give it a max concurrent stream count(set aside which of 4 or 6 are more reasonable).

While true, modern implementations use hardware blocks for acceleration AND for just terminals, with spare displays, you'll likely be using lower resolutions at lower refresh rates.

We've used a series of n150 based mini PCs (priced at $200 each) to power three dashboard displays at the same time. I guess it's a matter of calculating power usage and figuring out how that translates over 4 years

You kind of can do this already with a raspberry pi strapped to the back of a monitor and "barrier", the successor to synergy, to use the keyboard and pointing device of your laptop on as many displays on the same desk as you want. Each one is an independent computer.

https://github.com/debauchee/barrier

Unfortunately the new price of raspberry pi is kind of ridiculous.

Forget the terminal - do it with an FPGA!