News for hackers. Stuff that matters.
Re:Apache Kafka Goes 1.0 by connormcarthur on Wednesday November 01, 2017 @05:29PM (#15605633)
Do you find that you have to replay data often?

I'd imagine that 99% or more of that 400GB/day gets thrown away before ending up in Redshift.

Re:Apache Kafka Goes 1.0 by odammit on Wednesday November 01, 2017 @06:22PM (#15606039)
Rarely. We visualize a lot of it in Kibana and our kibana indexes are pruned after N days (configured for each event type).

We usually only replay when we want to restore data for querying stuff outside the pruning range.

We keep all that data forever.