News for hackers. Stuff that matters.

goyalanuj

Re:Data Infrastructure at IFTTT on Thursday October 15, 2015 @07:26PM (#10396615)
Whatever data we need in realtime, we do stream it to the Kafka cluster.

We don't do it for the production database because we don't need it in realtime.

Re:Data Infrastructure at IFTTT on Thursday October 15, 2015 @07:09PM (#10396555)
We have fixed it! Thank you!
Re:Data Infrastructure at IFTTT on Thursday October 15, 2015 @02:18PM (#10394914)
Thank you Jonathan!

As Jonathan mentioned, we made this decision around 9 months back and at that time Kinesis wasn't as mature and had less flexibility around retention period etc.

Kafka is very reliable (as I had seen it handling billions of events a day at LinkedIn) and has a huge open-source community around it. At IFTTT, we always prefer to use and contribute to open source ( http://engineering.ifttt.com/oss/2015/07/23/open-source/ ).