Usually you'd aggregate those logs to a centralised location and have some kind of visibility platform to parse them like Splunk, ELK, DataDog, etc. That same pipeline would also typically share at least part of the process for streaming your logs to a tamper proof archive.

So it's not that checking logs is a dying habit, it's process of SSHing onto the server and running shell commands to query them.

Though I'm not suggesting the latter is bad practice. In some domains it's suboptimal but in other domains it is a pretty reasonable response to fault diagnosis. IT is a broad field :)

>IT is a broad field :)

Yes it is. Today's assumption seems to be that everything is being done "web scale" with hundreds of servers in multiple datacenters across the globe.

I'm not so fortunate/unfortunate (depending on one's viewpoint) of having to deal with that kind of scale. I have less than 10 servers I maintain so a lot of these new fancy viewers is overkill for me.

I'm not going to judge how you do your job because I clearly don't know anything about the problems you're tasked to solve. However I will disagree that visibility is a web scale problem. I've still found having a centralised visibility platform useful even when managing a very small number of servers. After all, you need to set up monitoring and (presumably) alerts when something fails anyway. So it's just another service on your Nagios (for example) server.

Plus if you have any kind of auditing and compliance where you need to retain tamper proof logs for any period of time then you've already done the other half the work by streaming those logs off the servers.

> I have less than 10 servers I maintain so a lot of these new fancy viewers is overkill for me.

There are several layers to the complexity of "web scale orchestration" however, and you definitely can get the same tooling/features on a much smaller scale with less complexity involved.

I.e. you can rent a few baremetal servers from hetzner, give them an internal lan interface and setup k3s on that internal network. Now you can add capabilities as you want and need them: I.e.

  * deploying longhorn for a distributed storage backend with easily configured backup.

  * Certmanager for cluster wide https certificate provisioning 

  * Promstack to get healthchecks etc

  * Loki for the just now mentioned logging aggregation

Each of these capabilities are usually just one deployment command (kubetcl apply/helm install) and will enable this capability for all existing and future deployments on the same cluster.

If you have a weekend to experiment and want to explore what it actually feels to use that kind of tooling I can only encourage you to maybe try it out. The whole experience has gotten pretty nice over the years, and while it is pretty much the same you already know... Just differently, it does solve it in a very reusable way, which makes it ultimately (at least now after about 1 year of usage) much more straightforward to figure out then just a bunch of shell scripts

After I got over that it's done wrong, (as in: not what I've come to know/expect) I kinda got to like it, especially for small scale stuff (a 4 node HA cluster with 64gb ram each costs slightly below 200€ per month on hetzner, and they're using nvme storage, so really fast IO)

Even if you're not webscale, modern tooling is just so much better for visualizing website stats. Even if you have only one server, how do you make visualizations from the log file? You don't have to use something webscale and enterprise-grade like Splunk. I do recommend finding a modern tool that you like though. Something better than `awk`-ing a log file into CSV, loading that into a spreadsheet, and then fiddling with it to get a graph.

Not everything you can SSH into that produces logs is necessarily a "server". Small embedded systems may not have the RAM or flash space to aggregate locally, and setting them up for remote logging may be problematic or at least inconvenient for one reason or another.

> Not everything you can SSH into that produces logs is necessarily a "server".

I never suggested that was the case. I just don’t consider then in scope for this topic.

I have managed numerous such devices over my career. In fact most of my home networking gear would fall into that category of “produces logs and SSH but not a server” but none of which runs anything UNIX-like (so you couldn’t install this tool even if you wanted to). However they do support a bunch of different tooling for pushing logs and metrics over the network. Which then brings me back to my earlier point about log collection.

Also if we take your example of embedded devices with low memory and storage, then installing additional utilities like this one when you can already do the same thing (albeit less conveniently) with busybox probably isn’t a good idea. So they’d be out of scope for this thread too.

I’m not going to say that there’s no such thing as a non-server platform that produces logs that this tool would be useful for (I don’t believe in absolutes!) however it is reasonable to assume that servers and desktops, ie more classical computing devices, are a vastly larger demographic for this tool. Hence why I focused my examples around that.

As an embedded developer I have different view. Linux is everywhere. I've been working within nothing but embedded Linux constantly since 2005.

I regularly install debugging tools onto target devices (strace, tcpdump, ...) even though those don't go into a production image.

Currently, I'm using pw on an embedded system, for capturing detailed traces from an application. It's very useful because it decouples the rate of the traces from the serial port, while still letting me get the info I need out of it.

The stripped executable is tiny: 20 kilobytes. It's so small I'm stashing it in a configuration flash partition which survives re-flashes of the kernel and filesystem, so I have the utility there no matter which image I load onto the target.

The alternatives these days is examining the combined logs in an aggregator like Splunk or ELK.

We use a log aggregator and watch logs from hundreds of servers in a searchable & filterable interface like Kibana. We also use Loki and Elastic extensively. If you have money to spend, I think you'd be looking at Splunk or Datadog.

As a matter of fact, we do not have SSH access to our servers. Everything happens over a web interface that has been secured.