News for hackers. Stuff that matters.
Re:Monitoring demystified: A guide for logging, tracing, met... by say_it_as_it_is on Friday July 31, 2020 @05:25AM (#24007037)
Is there an open source solution for processing streams of structured and unstructured logs and routing then onward? I see solutions for moving logs to elastic or Kafka but nothing for evaluating the log.
Re:Monitoring demystified: A guide for logging, tracing, met... by ekimekim on Friday July 31, 2020 @11:28AM (#24009593)
This is a problem that is both solved again and again, but also all the available solutions are bad.

In my experience what happens is:

1. you start with a "ship logs from X to Y" product

2. you add more sources and more destinations, making it more of a central router. you add config options for specifying your sources and dests.

3. since the way you checkpoint or consume or pull or push certain sources or dests doesn't generalize, you end up buffering internally to present a unified "I have recieved / sent this message successfuly" concept to your inputs and outputs.

4. you want to do some basic transforms on the logs as you go. you implement "filters" or "transforms" or "steps" and make them configurable. your config now describes a graph of sources -> filters -> dests

5. your filters need to be more flexible. you add generic filters whose behaviour is mostly controlled by their config options. your configs grow more complicated as you use multiple layers of differently-configured filters

6. you have a bad turing complete programming language embedded in your config file. getting simple tasks done is possible, getting complex tasks done becomes an awful, inefficient and unreadable mess.

My solution to this cycle has been to just write simple hard-coded applications that can only do the job I need them to do. If they need a different configuration later I edit the source. I'm writing my transforms in a real programming language and I avoid the additional complexity of abstractions. Of course, that comes with its own costs but I consider it well worth it.

Re:Monitoring demystified: A guide for logging, tracing, met... by onefuncman on Friday July 31, 2020 @07:08AM (#24007511)
The "OG" in the space is collectd, which is still my favorite choice if you are responsible at the operating system level: https://collectd.org/wiki/index.php/Chains

https://github.com/elastic/logstash was one of the first modern approaches. I started using it less the more often I ran into JRuby related bugs.

https://github.com/trivago/gollum is my pick from the golang ecosystem.

There are many more variants depending on how much complexity you are trying to apply. If you need to apply machine learning models, for example, you're probably going to end up with something similar to Apache Storm, though I don't know if it's operational story has improved enough to consider it over other alternatives, I lost track years ago between Apache Spark and the half dozen other stream processing projects.

Re:Monitoring demystified: A guide for logging, tracing, met... by dig1 on Friday July 31, 2020 @06:51AM (#24007414)
Riemann [1]. You can create custom endpoint for accepting almost any kind of messages, logs or data.

[1] http://riemann.io/

Re:Monitoring demystified: A guide for logging, tracing, met... by EricE on Friday July 31, 2020 @12:24PM (#24010290)
>Is there an open source solution for processing streams of structured and unstructured logs and routing then onward?

https://securityonion.net

It doesn't route them onward - it will collect, aggregate and provide you the tools to correlate/analyze logs across your environment. Enable the built in network monitoring tools too and you have not only a powerful tool to help you with application management, but security as well (hence its namesake).

Beware - in pealing back the layers of your environment you can really get sucked in. I never seem to have enough hardware to do what I want with SO but it's pretty amazing what you can do with it.

EDIT - wow, I'm a little shocked that no one else has brought Security Onion up. I guess they need to up their advertising game!

Re:Monitoring demystified: A guide for logging, tracing, met... by gregwebs on Friday July 31, 2020 @11:24AM (#24009536)
You can actually do this log manipulation in fluent-bit (you can write Lua if you need to) although the forwarding cannot be routed to different locations.
Re:Monitoring demystified: A guide for logging, tracing, met... by a10c on Friday July 31, 2020 @10:49AM (#24009109)
https://vector.dev/ sounds pretty close
Re:Monitoring demystified: A guide for logging, tracing, met... by malechimp on Friday July 31, 2020 @05:34AM (#24007085)
Maybe you're looking something like this https://docs.tremor.rs/
Re:Monitoring demystified: A guide for logging, tracing, met... by ysoft on Friday July 31, 2020 @06:53AM (#24007422)
I haven't found anything, we are moving to hosted Humio really soon. It uses kafka