What are you doing that makes a half-terabyte of logs per day? That seems kinda nuts.
Ads ads ads!
That’s just one system. Our Nginx access logs alone are about 250GB per day (they use that system).
Re:Apache Kafka Goes 1.0
by connormcarthur on Wednesday November 01, 2017 @05:29PM
(#15605633)
Do you find that you have to replay data often?
I'd imagine that 99% or more of that 400GB/day gets thrown away before ending up in Redshift.
Rarely. We visualize a lot of it in Kibana and our kibana indexes are pruned after N days (configured for each event type).
We usually only replay when we want to restore data for querying stuff outside the pruning range.
We keep all that data forever.
If we need to replay something we just move the group of files in and out of the folder on s3 (bucket/logs -> bucket/temp -> bucket/logs) and it triggers a new objectCreated and fires the lambda.
We execute the 2 move operations with the AWS cli tool.
The one other thing we do is we have a config file in bucket/config.json that’s says event type X should go to redshift, pg, es, where ever. So we can tweak that before a move to send data to additional data stores.
We follow ELT instead of ETL for streaming into data stores.