Re:Billions of Messages a Day – Yelp's Real-Time Data Pipeline
by TeMPOraL on Saturday July 16, 2016 @05:37AM
(#12105785)
Whenever I see headlines like these, "X billion messages per day/hour/second in some web service Y", I wonder what the hell is that service doing that it generates so much messages? I.e. I could understand Facebook, with its billion+ of users and built-in messaging platform, could generate billion of "messages" a day. But a mostly read-based service like Yelp?
But I finally realized - those messages are probably mostly tracking, ads, more tracking, some infrastructure work and even more ads & tracking. The sausage machine that turns people into money.
Re:Billions of Messages a Day – Yelp's Real-Time Data Pipeline
by patrickmay on Saturday July 16, 2016 @10:23AM
(#12106405)
Indeed. I'm working in adtech and we get four billion requests for bids per day (more on Black Friday and the holiday season). Add in the ad serving and processing of all the results and it blows up fast.
Re:Billions of Messages a Day – Yelp's Real-Time Data Pipeline
by eellpp on Saturday July 16, 2016 @12:09AM
(#12105199)
Applying the 80/20 principle: 80% of traffic occurs in the 20% of years time. Which is around 12 reviews per second. Still modest.
Re:Billions of Messages a Day – Yelp's Real-Time Data Pipeline
by NhanH on Saturday July 16, 2016 @01:37AM
(#12105370)
Not entirely on topic, but is the 80/20 principle recursive?
Re:Billions of Messages a Day – Yelp's Real-Time Data Pipeline
by thesimpsons1022 on Saturday July 16, 2016 @09:26PM
(#12108610)
20 percent of it is 80 percent recursive
Re:Billions of Messages a Day – Yelp's Real-Time Data Pipeline
by nsp on Saturday July 16, 2016 @03:05AM
(#12105548)
Yes.
Re:Billions of Messages a Day – Yelp's Real-Time Data Pipeline
by ju-st on Saturday July 16, 2016 @05:59AM
(#12105821)
Even a Sqlite DB could handle this
Re:Billions of Messages a Day – Yelp's Real-Time Data Pipeline
by discordance on Saturday July 16, 2016 @07:46PM
(#12108327)
Also, 80% (often 90%) are reads and 10-20% are writes.
Re:Billions of Messages a Day – Yelp's Real-Time Data Pipeline
by nirmel on Friday July 15, 2016 @05:09PM
(#12103858)
100 million is cumulative, not annual.
Re:Billions of Messages a Day – Yelp's Real-Time Data Pipeline
by falsedan on Friday July 15, 2016 @08:15PM
(#12104608)
> 100 million reviews/year is only 3 reviews per second on average
That's writes, I imagine the reads would be a bit higher.
Re:Billions of Messages a Day – Yelp's Real-Time Data Pipeline
by lumpypua on Friday July 15, 2016 @09:18PM
(#12104777)
And ultimately, writes resulting from those reads. I imagine yelp has pretty stout analytics/metrics pipeline fed off views and conversion tracking.
Re:Billions of Messages a Day – Yelp's Real-Time Data Pipeline
by rlau26 on Saturday July 16, 2016 @12:15AM
(#12105213)
This. A lot of people look at apps like Instagram and Facebook, and think that they're simple to build and manage, when they don't realize that the part of the app the consumer directly interacts with on their screen is the tip of the iceberg when it comes to the entire business.
Reading between the lines it seems the problem wasn't scaling, but programmer productivity. Smaller code bases is often easier to work with so I guess they solved that by dividing it up into many small services. The blog could use a more detailed description of the problem they are actually solving.