News for hackers. Stuff that matters.
Re:AppLovin: Processing 30B Requests A Day by apple-fan on Friday March 13, 2015 @08:06PM (#9200867)
Thank you! Let me try to address those questions:

> 1. Is the Spark cluster running on top of an in-house cluster or on AWS EMR/S3?

It runs on an in-house cluster.

> 2. Any more details about the custom message system, specifically how to guarantee exactly once delivery?

This topic might deserve a separate article. We have built a custom locking system that ensures proper delivery.

> 3. How do you guarantee data is never dropped?

We write each data point into 2-3 locations in our system and also back it up in S3.

> 4. "Make sure every message can be re-played without data corruption" how to achieve this when there is so many data points coming in?

Basically, batch things and keep a lot of signatures and checksums around.

Re:AppLovin: Processing 30B Requests A Day by alstream on Friday March 13, 2015 @09:49PM (#9201222)
Thanks for your time!

About point 4, I'm assuming you built in idempotency into your message replay system. Is your unit of idempotency, say, 1 hour or some other unit of time, or is every single message have idempotent results? Because the latter would be pretty amazing :)