Re:Cache made consistent: Meta’s cache invalidation solution
by uvdn7 on Wednesday June 08, 2022 @04:54PM
(#31673969)
For the fun issues you described, read-modify-write (either using optimistic concurrency control or pessimistic) can work, if I understand your question correctly.
Spanner is awesome and one of my favorite systems/papers. I think it would be very computational and power expensive to run the social graph workload on a spanner-like system. Do you have an estimate of how many megawatts (if not more) are needed to support one quadrillion queries a day on Spanner?
Re:Cache made consistent: Meta’s cache invalidation solution
by benlivengood on Wednesday June 08, 2022 @07:08PM
(#31675611)
I wish I had more solid data. Cloud Spanner claims about 10K read QPS and 2K write on a "node" which costs $1/hour. The Spanner paper reports about 10K read QPS per core. As I understand the cloud deployment it's 3 replicas at that price, so $0.33/hour should buy me about 8 cores and so I'm not sure what the disparity is (maybe markup? CloudSQL is 2X the cost of compute), but I'll go with 10K QPS/8 cores at the low end. Anyway, something like 500-1000W for ~100 cores so something between 100 QPS/W and 10 QPS/W using the 1000W and high and low performance estimates, or 10^15 / 86400 = 11.6e12 QPS for somewhere between 116MW and a a little over a GW. Sounds comparable(?) to MySQL+TAO on the low end to ridiculously expensive at the high end.
EDIT: I have no clue how efficient MySQL+TAO really are but figure that at least tens of thousands of machines go into it.
Re:Cache made consistent: Meta’s cache invalidation solution
by folmar on Wednesday June 08, 2022 @07:38PM
(#31675880)
It that's on x86_64 than the power budget seems some ten times too low for me. A typical Xeon Silver 4208 is 10 W TDP per core, and other things would also add up, even taking into account that newer CPUs are more efficient.
Re:Cache made consistent: Meta’s cache invalidation solution
by benlivengood on Wednesday June 08, 2022 @07:52PM
(#31675982)
I am basing this off of Epyc Milan which looks closer to 5W/core, and possibly mutilating my language a bit because I think cloud products are sold by thread (vCPU), not by physical core.
I was actually a little surprised how hard it is to get performance/W figures since I don't currently have any physical server hardware to go off of.
Re:Cache made consistent: Meta’s cache invalidation solution
by uvdn7 on Thursday June 09, 2022 @10:35AM
(#31681691)
I can't give away details/numbers. But I have to say, what you did right here is very impressive.
Re:Cache made consistent: Meta’s cache invalidation solution
by DeathArrow on Thursday June 09, 2022 @02:17AM
(#31678066)
>Do you guarantee referential integrity in TAO?
Since I observed data inconsistencies on Facebook multiple times, I believe they aren't as concerned about data integrity as a company doing financial transactions would be.
Contrasting that with Spanner where cache coherency is solved by reading from the recent past at consistent snapshots or by issuing global transactions that must complete without conflict within a brief truetime window. I am guessing the cost of Spanner underneath the social graph would be a bit too much for whatever benefits might be gained, but curious if anyone looked into using something similar.