Re:Confluent, a company for Apache Kafka and realtime data
by beagle3 on Friday November 07, 2014 @09:28AM
(#8572597)
Both this solution (and easytiger's above) work if disconnects happen rarely and for short times.
But it my case, I have "30 minutes on, 5 minutes off, 90 minutes on, 90 minutes off, 2 minutes on" kind of situations, in which anything that doesn't track what was already transferred and what wasn't, will lose data. (zeromq's buffers also have limited capacity and/or are tied to a process on the other side - if it restarts, buffers are gone).
kafka sounds like an enormous overkill. If you want to store the logs locally while tailing, just add in a tee.