News for hackers. Stuff that matters.
Re:Redis Streams and the Unified Log by hexsprite on Wednesday November 08, 2017 @11:52AM (#15654196)
I'm still pretty new to event sourcing but when I looked at Kinesis it seemed less than ideal for ES due to it's finite retention (max 168hrs). How do you get around that?
Re:Redis Streams and the Unified Log by squeaky-clean on Wednesday November 08, 2017 @12:22PM (#15654518)
My company uses Kinesis for stream consumption, but everything is stored in both Redshift and Glacier (we expire redshift data after a year or two but Glacier is kept forever). This is our canonical data store.

Smaller subsets of events are kept in Redshift/Postgres/MongoDB databases, depending on the purpose and how they'll be queried. These keep anywhere from 48 hours to 6 months of data depending on the purpose, sometimes it's just the raw events filtered in a certain way, and sometimes a projection based on those events. Either way, if we catch a bug in our logic we can re-run it for the past year of data or whatever.

Re:Redis Streams and the Unified Log by CaveTech on Wednesday November 08, 2017 @12:05PM (#15654352)
Kinesis is designed to be consumed, it's not a long-term data store. It's good for quickly shuffling data with some guarantees, and then you stream/batch process it in semi-realtime after the fact.