Fluentd is great, and works very well to inject logs into Elasticsearch. We were using it at OpenDNS. We didn't even try Logstash, as Fluentd did the job.

At OVH, both turned out to be way too slow, so we wrote Flowgger which is heavily used by all our services: https://github.com/jedisct1/flowgger

I'm not an elastic person, but I can shed some light on this: You're holding it wrong. It's not a bug. You have multiple config files in one directory - if you do that, all those files are combined to one, that means that each event gets handed to each of your individual outputs - multiplying the message. See https://www.elastic.co/guide/en/logstash/current/command-lin...

Feel free to hop on the IRC if you have further questions, there's usually somebody qualified to answer.

I appreciate the help, though isn't the point of `/etc//conf.d` directories generally that you have multiple config files? This is a common idiom that other packages handle correctly (differently?).

I have hopped on the logstash IRC at times to ask about some of this, though I guess not this exact item. In fact, there's a different (well-known issue) that the init script for logstash has the config path hard-coded: https://botbot.me/freenode/logstash/2015-11-17/?msg=54338903...

There's also the problem that logstash (and forwarder) doesn't seem to let me do anything useful with the file names. I could work around that, sure, but it would be nice to have meaningful file names (not the "ls." thing that LS uses). Syslog-ng, for comparison, gives you a lot of control of that.

> isn't the point of `/etc//conf.d` directories generally that you have multiple config files?

Yes certainly. It's totally fine to place multiple config files there, I do as well. I split up my configs in the various outputs, inputs etc. It's just that logstash combines them to a single pipeline and does not run a pipeline per config file. Nginx doesn't run a webserver per config file either :).

It's certainly something that's unexpected and could be much better documented, but alas, I'm just a user :)

(and I do agree, your issue could have been handled much better, especially since it's not actually a bug)

Awesome, thanks for the clarification. This might help (if I ever go back to using logstash at this point!).

Yeah, my point was more that they accepted my issue, but there's no action and there are more than 600+ other open issues. Seems Elastic is too busy branding and pushing breaking changes to their APIs.

I do sincerely appreciate your clarifications and comments on this one, though.

(I've never gotten used to HN's non-Markdown markup.)

Oh, well, interesting that I just notice your HN username is also that IRC username. I suppose you're the one that replied to me there! Thank you!

likely, I tend to use the same name all over the internet :) You're welcome.

Don't get too excited about Riemann. It's a stream processing engine, not data transport. It has its uses, but not for shipping logs.

syslog-ng and its competitor, rsyslog, are fine, but they are concentrated around logs specifically. Fluentd (and logstash) can be used to transport other data, too, like monitoring or inventory.

Good info (from jedisct1 as well) -- the distinctions are indeed important!