Re:Confluent, a company for Apache Kafka and realtime data
by mwarkentin on Friday November 07, 2014 @09:28AM
(#8572601)
Sounds like a job for logstash: http://logstash.net
If you don't mind using a 3rd party service, you could look into using Papertrail, Loggly, etc.
Re:Confluent, a company for Apache Kafka and realtime data
by beagle3 on Friday November 07, 2014 @09:37AM
(#8572634)
3rd party is not an option. I'll have to look at logstash, thanks.
Re:Confluent, a company for Apache Kafka and realtime data
by easytiger on Friday November 07, 2014 @09:08AM
(#8572523)
How about a tiny forwarder on each device. It fopens the file and reads to the end, when the file is appended to it will be able to continue reading. Packet up each line and forward to you server using zeromq PUSH/PULL. If you are disconnected they queue up and send when zeromq automagically reconnects.
Re:Confluent, a company for Apache Kafka and realtime data
by imperialWicket on Friday November 07, 2014 @12:33PM
(#8573601)
Since logstash was already mentioned, it's worth putting https://github.com/elasticsearch/logstash-forwarder here, as the logstash team built it as a solution for logstash-type needs on systems that may not be able to support logstash itself.
Re:Confluent, a company for Apache Kafka and realtime data
by gelliott on Friday November 07, 2014 @09:23AM
(#8572575)
Can your endpoints log to syslog so they can be forwarded on to a central collector?
Re:Confluent, a company for Apache Kafka and realtime data
by beagle3 on Friday November 07, 2014 @09:29AM
(#8572603)
Probably. Do you know of a syslog/forwarder that deals properly with intermittent connections? (i.e., available for an hour, then gone for an hour, then available again, etc.)
Re:Confluent, a company for Apache Kafka and realtime data
by b0ti on Friday November 07, 2014 @12:29PM
(#8573581)
Not sure about logstash but a lot of its users use NXLog (http://nxlog.org) as a shipper since it has a much lower resource footprint as there is no java and ruby in there. (I'm affiliated with NXLog.)
Re:Confluent, a company for Apache Kafka and realtime data
by otterley on Friday November 07, 2014 @02:13PM
(#8574125)
Syslog-NG PE works well for this. It can be configured to use a per-destination disk buffer, so that if the destination goes offline, messages will queue until they can be sent again.
Re:Confluent, a company for Apache Kafka and realtime data
by lrm242 on Friday November 07, 2014 @09:16AM
(#8572541)
autossh + ssh -C + tail -F ?
kafka sounds like an enormous overkill. If you want to store the logs locally while tailing, just add in a tee.
Re:Confluent, a company for Apache Kafka and realtime data
by beagle3 on Friday November 07, 2014 @09:28AM
(#8572597)
Both this solution (and easytiger's above) work if disconnects happen rarely and for short times.
But it my case, I have "30 minutes on, 5 minutes off, 90 minutes on, 90 minutes off, 2 minutes on" kind of situations, in which anything that doesn't track what was already transferred and what wasn't, will lose data. (zeromq's buffers also have limited capacity and/or are tied to a process on the other side - if it restarts, buffers are gone).
There's a missing piece in the realtime puzzle, at least one that I haven't been able to find, for which Kafka is an overkill - perhaps someone here knows of a solution:
I have tens of system endpoints connected through unreliable (Line-of-sometimes-occluded-sight, 2G and 3G WWAN, some are in vehicles so connections are intermittent).
I just want to consistently tail their logs in a bandwidth-efficient, connection-drop resistant way; and I can't find any standard thing that does this.
Kafka would fit the bill in general, but would require a lot of work (reading textual logs into kafka, querying kafka for new stuff across connection, reading from kafka and writing to text files) - and I'm not sure how well it deals with dropped connections.
My existing solution is to rsync the log directories (--append, --inplace) as infrequently as I can from an operational view, which is 1 minute. It is relatively bandwidth efficient (although could be much better), robust with respect to connection issues, and generally works.
However, it is less efficient than it could be: if directories have a lot of files, like /var/log often does, there's a lot of sync overhead. The delay is 1 minute instead of a couple of seconds (which is what you would get with a simple "tail -f" through a TCP connection), and it doesn't play well with common log rotation schemes (though that's relatively easy to work around).
Anyone has a better solution, kafkaesque or otherwise?