Re:Redis Streams and the Unified Log
by hexsprite on Wednesday November 08, 2017 @11:52AM
(#15654196)
I'm still pretty new to event sourcing but when I looked at Kinesis it seemed less than ideal for ES due to it's finite retention (max 168hrs). How do you get around that?
[ Parent
| on HN ]
Re:Redis Streams and the Unified Log
by squeaky-clean on Wednesday November 08, 2017 @12:22PM
(#15654518)
My company uses Kinesis for stream consumption, but everything is stored in both Redshift and Glacier (we expire redshift data after a year or two but Glacier is kept forever). This is our canonical data store.
Smaller subsets of events are kept in Redshift/Postgres/MongoDB databases, depending on the purpose and how they'll be queried. These keep anywhere from 48 hours to 6 months of data depending on the purpose, sometimes it's just the raw events filtered in a certain way, and sometimes a projection based on those events. Either way, if we catch a bug in our logic we can re-run it for the past year of data or whatever.