News for hackers. Stuff that matters.
Re:Show HN: My Single-File Python Script I Used to Replace S... by xorcist on Thursday September 21, 2023 @02:09PM (#37601568)
Fetching logs regularly sounds hard? Wouldn't you need to keep track of the position of all files, with heuristics around file rotations? And if something catastrophic happens, the most interesting data would be in that last block which couldn't be polled?

Normally you'd avoid all that complexity by shipping logs the other way, sending from each machine. That way you can keep state locally should you need to. All unix-like systems do this out of the box, and almost all software supports the syslog protocol to directly stream logs. But you can also use something like filebeat and a bunch of other modern alternatives.

The analyzer can then run locally on the log server and a whole lot of complexity just disappears.

Re:Show HN: My Single-File Python Script I Used to Replace S... by eigenvalue on Thursday September 21, 2023 @02:52PM (#37602238)
I considered doing it the way you described, but then you need to deploy software on every single one of your machines and make sure it's running, that it's not accidentally using up 99% of your CPU (I've had bad experiences with the monitoring agents for Splunka and Netdata misbehaving and slowing down the machines and causing problems), etc. Whereas with the "pull" approach I used in my tool, you don't need to deploy ANY software to the machines you are monitoring-- you just connect with SSH and grab the files you need and do all the work on your control node.
Re:Show HN: My Single-File Python Script I Used to Replace S... by xorcist on Thursday September 21, 2023 @03:16PM (#37602655)
One way or the other, your hosts are running your application, and you are already deploying software on every single host. But I hear you with some of the agents. That's why I mentioned syslog.

It's already there, it's supported by most logging packages, and it's dead simple. No additional software required. All text. What it doesn't do is structured logging, but analyzing on the log host is often enough.

Agents aren't all that bad however, and you're likely already running some agent like icinga or zabbix for regular monitoring.

Re:Show HN: My Single-File Python Script I Used to Replace S... by hjvalente on Friday September 22, 2023 @05:50AM (#37609998)
having a Netdata agent taking your machine's CPU to 99% shouldn't happen, not sure when was the list time you tried it but a lot of recent improvements have been done on the Netdata Agent

also, with Netdata you can achieve the same architecture design using a Netdata Parent that could be your "control node" and to where you stream the metrics of the nodes you want to keep running with as less load as possible - you can even offload the health engine and the machine learning

take a look at https://learn.netdata.cloud/docs/streaming/ and https://learn.netdata.cloud/docs/configuring/how-to-optimize...

Re:Show HN: My Single-File Python Script I Used to Replace S... by eigenvalue on Friday September 22, 2023 @01:29PM (#37614935)
I hear you, but once something like that burns me, I am very loathe to risk it again if the cost/benefit ratio seems unfavorable to me. While it's nice to have the pretty dashboards for Netdata, it's not worth even a small risk of it breaking or degrading my systems.
Re:Show HN: My Single-File Python Script I Used to Replace S... by johnthescott on Friday September 22, 2023 @07:05PM (#37618934)
> Wouldn't you need to keep track of the position of all files

rsync --append is your friend.

Re:Show HN: My Single-File Python Script I Used to Replace S... by xorcist on Tuesday September 26, 2023 @08:27AM (#37658271)
That doesn't take neither rotated nor truncated log files into account. The easiest way, and what most log shippers do, is following inodes.

It's also not as effective as streaming them to their intended target directly. Syslog can write a complementary local copy too should you wish to keep one.

Logs has been a thing since the past forty years. In order to reinvent it, it is good to be acquainted with the standard systems.