News for hackers. Stuff that matters.
Re:Show HN: My Single-File Python Script I Used to Replace S... by hliyan on Thursday September 21, 2023 @01:07PM (#37600643)
A long long time ago, I used a series of tail -f's and unix pipes to aggregate logs, and grep, less and awk to analyse them. There were about 20 different services written in C++, each producing over 1GB of logs each day. Managed to debug some fairly complex algorithmic trading bugs. Twenty years later, I still can't fathom why we're spending so much money on Splunk, DataDog an the like.
Re:Show HN: My Single-File Python Script I Used to Replace S... by dsXLII on Thursday September 21, 2023 @01:16PM (#37600773)
Volume. 1GB of data per day is rounding error. If you have tens of thousands of servers, each generating hundreds of gigabytes of data per day, tail -f and grep don't scale especially well.
Re:Show HN: My Single-File Python Script I Used to Replace S... by getrealyall on Thursday September 21, 2023 @01:28PM (#37600963)
And I bet a hang glider can't fly from New York to Paris, either! The nerve!

Recall that the poster said this was for a small startup. If you're Google, by all means, use Google logging tools. If you aren't, then solve the problem you have, not the problem your résumé needs.

Re:Show HN: My Single-File Python Script I Used to Replace S... by phyrex on Thursday September 21, 2023 @01:46PM (#37601260)
The guy asked

> Twenty years later, I still can't fathom why we're spending so much money on Splunk, DataDog a the like.

And the poster above answered that question

Re:Show HN: My Single-File Python Script I Used to Replace S... by vincnetas on Thursday September 21, 2023 @01:25PM (#37600907)
100GB of logs per day? what kind of applications are that chatty?
Re:Show HN: My Single-File Python Script I Used to Replace S... by jononomo on Thursday September 21, 2023 @02:00PM (#37601472)
Yeah, the solution here is to get rid of 98% of the logging.
Re:Show HN: My Single-File Python Script I Used to Replace S... by moqmar on Friday September 22, 2023 @02:09AM (#37608458)
Probably Java/JVM... Never seen something where all kinds of libraries log more.
Re:Show HN: My Single-File Python Script I Used to Replace S... by vincnetas on Friday September 22, 2023 @11:19AM (#37613186)
Log level configuration is a cheap solution in this case.
Re:Show HN: My Single-File Python Script I Used to Replace S... by 10000truths on Thursday September 21, 2023 @02:37PM (#37602023)
They scale perfectly fine, as long as you filter locally before aggregating. Lo and behold:

  mkdir -p /tmp/ssh_output
  while read ssh_host; do
      touch "/tmp/ssh_output/${ssh_host}.log"
      ssh "$ssh_host" grep 'keyword' /var/my/app.log > "/tmp/ssh_output/${ssh_host}.log" &
  done < ssh_hosts.txt
  wait
  cat /tmp/ssh_output/*.log
  rm -rf /tmp/ssh_output
Tweak as needed. Truncation limits and real-time tailing are left as an exercise to the reader.
Re:Show HN: My Single-File Python Script I Used to Replace S... by getrealyall on Thursday September 21, 2023 @01:32PM (#37601034)
Financialization and mediocre developers. I haven't worked with too many people I could actually trust to even emit logs correctly, let alone develop a tool to collect and aggregate them.

I've also been told, time and again, in no uncertain terms, to "buy as much as possible". We've reached the logical conclusion of SaaS-everything: every company just cobbles together expensive, overcomplicated computers from other expensive, overcomplicated computer providers, resulting in expensive, bloated systems that barely work.

Re:Show HN: My Single-File Python Script I Used to Replace S... by diarrhea on Thursday September 21, 2023 @03:15PM (#37602640)
Buying everything and SaaSing the whole place up is a true killjoy. I giggle with joy whenever I am allowed to write code. And then a support request comes in that I get assigned to, “thing in SaaS doesn’t work please fix”. And all you have to debug that SaaS is their UI. The checkbox in question is on, you notice, so it can only be a bug on their side. Off to contacting support as the only available avenue. Incredibly boring.
Re:Show HN: My Single-File Python Script I Used to Replace S... by quickthrower2 on Friday September 22, 2023 @02:48AM (#37608714)
Yay I get to write code! Oh it’s Pulumi code that wraps terraform to deploy Elastic Search and configure networking to allow logs from a kubernetes we deployed the same way.
Re:Show HN: My Single-File Python Script I Used to Replace S... by fragmede on Thursday September 21, 2023 @01:55PM (#37601385)
What did you use for visualization in that stack? The fact that I can "|" (pipe) my data and make bar and pie charts is what really does it for me. What's really money is trivially being able to see requests coming in overlaid on a world map. I was sold the first time I saw that because it let me fix an issue that would have taken me hours to suss out just grepping around.

More power to you for using sed awk and grep, they're powerful tools and every computer person should know how to use them. But if you're hung up on only using sed awk and grep for emotional reasons, that's self-limiting. We have better tools today, and you don't get hero points for using shittier tools when there are better ones available to you.

https://www.splunk.com/en_us/blog/tips-and-tricks/mapping-wi...