I'm not going to judge how you do your job because I clearly don't know anything about the problems you're tasked to solve. However I will disagree that visibility is a web scale problem. I've still found having a centralised visibility platform useful even when managing a very small number of servers. After all, you need to set up monitoring and (presumably) alerts when something fails anyway. So it's just another service on your Nagios (for example) server.

Plus if you have any kind of auditing and compliance where you need to retain tamper proof logs for any period of time then you've already done the other half the work by streaming those logs off the servers.

> I have less than 10 servers I maintain so a lot of these new fancy viewers is overkill for me.

There are several layers to the complexity of "web scale orchestration" however, and you definitely can get the same tooling/features on a much smaller scale with less complexity involved.

I.e. you can rent a few baremetal servers from hetzner, give them an internal lan interface and setup k3s on that internal network. Now you can add capabilities as you want and need them: I.e.

  * deploying longhorn for a distributed storage backend with easily configured backup.

  * Certmanager for cluster wide https certificate provisioning 

  * Promstack to get healthchecks etc

  * Loki for the just now mentioned logging aggregation

Each of these capabilities are usually just one deployment command (kubetcl apply/helm install) and will enable this capability for all existing and future deployments on the same cluster.

If you have a weekend to experiment and want to explore what it actually feels to use that kind of tooling I can only encourage you to maybe try it out. The whole experience has gotten pretty nice over the years, and while it is pretty much the same you already know... Just differently, it does solve it in a very reusable way, which makes it ultimately (at least now after about 1 year of usage) much more straightforward to figure out then just a bunch of shell scripts

After I got over that it's done wrong, (as in: not what I've come to know/expect) I kinda got to like it, especially for small scale stuff (a 4 node HA cluster with 64gb ram each costs slightly below 200€ per month on hetzner, and they're using nvme storage, so really fast IO)

Even if you're not webscale, modern tooling is just so much better for visualizing website stats. Even if you have only one server, how do you make visualizations from the log file? You don't have to use something webscale and enterprise-grade like Splunk. I do recommend finding a modern tool that you like though. Something better than `awk`-ing a log file into CSV, loading that into a spreadsheet, and then fiddling with it to get a graph.