News for hackers. Stuff that matters.
Re:Scaling Financial Reporting at Airbnb by d--b on Thursday March 16, 2017 @06:44PM (#13889843)
Is it really that big that we need to talk about "scaling" it? 4-5 hours of processing time is a lot... I mean how many transactions would they do on a normal day?
Re:Scaling Financial Reporting at Airbnb by dangoldin on Thursday March 16, 2017 @10:11PM (#13890817)
Yea - it does seem a bit high. We use Spark for our adtech data pipeline and we're handling tens of billions of events a day in less time. It may be a function of how much data they're pulling in from other systems or dumping the data back into a variety of systems. Spark itself is parallelizable so in theory can be sped up just by running more nodes.
Re:Scaling Financial Reporting at Airbnb by sheeshkebab on Friday March 17, 2017 @01:14PM (#13895798)
financial processing is typically sequential - can't calculate some metric until some other thing was calculated (or pulled data for)... not well parallelizable in other words. or so it is with some systems I deal with.