Using logs to build a solid data infrastructure

Re:Using logs to build a solid data infrastructure by Sphax on Wednesday May 27, 2015 @06:20PM (#9614805)

There's a log compaction cleanup policy yes. Never used it myself but if I'm not mistaken it works like this: for each message you send to Kafka, you set a key with it. When Kafka does log compaction, it keeps only the last value for each key.

The other cleanup policy is to just have a retention time. After X minutes/days/weeks segments of the log are simply deleted.

[ Parent | on HN ]

Re:Using logs to build a solid data infrastructure by moatra on Wednesday May 27, 2015 @06:30PM (#9614866)

That sounds great if your messages in the logs are the complete state for that key, but I'm not seeing how to use that compaction system if the messages are change events.

Is there a system designed for snapshotting the aggregate and logging the delta?

[ Parent | on HN ]

Re:Using logs to build a solid data infrastructure by kasey_junk on Wednesday May 27, 2015 @08:07PM (#9615266)

A common pattern is to publish a "checkpoint" message. Not sure if the concept is built into Kafka or not.

[ Parent | on HN ]

Re:Using logs to build a solid data infrastructure by erichmond on Wednesday May 27, 2015 @08:08PM (#9615272)

It's easy to store messages in HDFS or S3 for long-term storage. It's also easy to replay messages from those mediums, if you need to re-ingest data later on.

[ Parent | on HN ]