Re:AppLovin: Processing 30B Requests A Day
by themartorana on Tuesday March 10, 2015 @07:22AM
(#9177106)
Although it's difficult to fully articulate, as our traffic is basically a bell curve through the day and we auto scale between say 7 and 13 servers throughout the day... We pay about $4200/month. That includes several TB of traffic (which we gzip compress where it makes sense).
Re:AppLovin: Processing 30B Requests A Day
by alexatkeplar on Tuesday March 10, 2015 @08:32AM
(#9177328)
Thanks for sharing this - it's great to see this level of detail on an adtech stack. I wanted to ask a little more about the user profiles in Aerospike. What process is writing 12k profiles a sec, and what's the read path - is Aerospike e.g. wired into Nginx using https://github.com/aerospike/aerospike-mod-lua ? Any gotchas with Aerospike versus e.g. a Cassandra or DynamoDB?
Re:AppLovin: Processing 30B Requests A Day
by apple-fan on Tuesday March 10, 2015 @05:21PM
(#9180726)
We don't really use Lua: we connect to Aerospike directly, using a C driver. This saves a lot of time and allows sub-1ms responses.
I find Aerospike latency a lot more stable compared to that of Cassandra. Our use case is quite simple: we treat Aerospike as a large and fast key-value storage.
Re:AppLovin: Processing 30B Requests A Day
by alstream on Friday March 13, 2015 @09:49PM
(#9201222)
Thanks for your time!
About point 4, I'm assuming you built in idempotency into your message replay system. Is your unit of idempotency, say, 1 hour or some other unit of time, or is every single message have idempotent results? Because the latter would be pretty amazing :)
Re:AppLovin: Processing 30B Requests A Day
by coloneltcb on Monday March 09, 2015 @08:09PM
(#9175018)
Hi kumarm - Michael from AppLovin here. Thanks for the comment, it's an important concern. Here is a blog post outlining the problem and corrective actions that were taken: http://blog.applovin.com/applovin-security-notice/
Re:AppLovin: Processing 30B Requests A Day
by apple-fan on Tuesday March 10, 2015 @02:52PM
(#9179785)
We use a combination of software (ipvs, haproxy) and hardware load balancers. We plan on gradually moving away from traditional hardware load balancers.
Ex: Realtime stats system. On each request doing several get,set,increment,push-into-queue, geoip + 1 bulk-postgresql-upsert every second. 1ms response, 400 requests/second, 3 python processes, 10$/month shared hosting.
FreeBSD 10 on the EC2 boxes, whatever Linux AWS is running on their managed service instances. Nginx backed by Python and Go servers.
Plus a smattering of other services like ELBs, etc.
I can't imagine having something so critical that you build a custom Nginx plugin. I thought I was cool for doing a bit of Lua scripting. :)
I find Aerospike latency a lot more stable compared to that of Cassandra. Our use case is quite simple: we treat Aerospike as a large and fast key-value storage.
1. Is the Spark cluster running on top of an in-house cluster or on AWS EMR/S3?
2. Any more details about the custom message system, specifically how to guarantee exactly once delivery?
3. How do you guarantee data is never dropped?
4. "Make sure every message can be re-played without data corruption" how to achieve this when there is so many data points coming in?
Again, thank you for the insights!
> 1. Is the Spark cluster running on top of an in-house cluster or on AWS EMR/S3?
It runs on an in-house cluster.
> 2. Any more details about the custom message system, specifically how to guarantee exactly once delivery?
This topic might deserve a separate article. We have built a custom locking system that ensures proper delivery.
> 3. How do you guarantee data is never dropped?
We write each data point into 2-3 locations in our system and also back it up in S3.
> 4. "Make sure every message can be re-played without data corruption" how to achieve this when there is so many data points coming in?
Basically, batch things and keep a lot of signatures and checksums around.
About point 4, I'm assuming you built in idempotency into your message replay system. Is your unit of idempotency, say, 1 hour or some other unit of time, or is every single message have idempotent results? Because the latter would be pretty amazing :)
Any comment on that?