Re:“[Twitter] is doing 1000 poorly batched RPCs just to rend...
by csande17 on Sunday November 13, 2022 @07:27PM
(#33588876)
I suspect most of Twitter's performance issues are of the too-many-cooks variety, where services are poorly coupled or have a lot of dependencies because a million different teams are working on each one. (Heavy layoffs are a good step towards fixing that, along with a technical redesign.)
Re:“[Twitter] is doing 1000 poorly batched RPCs just to rend...
by ilaksh on Sunday November 13, 2022 @03:22PM
(#33586943)
I suspect this is not quite what that sounds like (possibly slightly misunderstood), really about the complexity of the task and the amount of data points, and human nature than it is some gross engineering mistake.
The engineers brought in are in the position where they are expected to trash the existing teams code. And then in the context of a slow page load, they might look at it and see up to 1000 of something in RPC and tell Musk and it comes out in a slightly mixed up way when he tries to repeat it.
The reason the number of calls gets high is (I guess) not from 1000 individual different types of requests, but from pulling many pages or records each of several things.
Anyway it is easy to trash other people's code especially if that's what your boss wants you to do to justify bringing you aboard. I have been the victim of it and also on one or two occasions done the same thing myself to some degree.
Another aspect is that no engineer really wants to maintain someone else's code if they don't have to. They are going to have their own viewpoint on architecture etc. and coming onboard is a prime opportunity to start a rewrite. So slamming the existing architecture is just an easy way to try to get that authorized.
I think it's a bit of an integrity test coming in and talking about someone else's code base. But people are unfortunately creatures of circumstance, and these things can bring out the worst in some as far as fairness goes. But again, even for a fair person it might be easy in that position to scoff at a codebase.
Most people are not particularly high integrity though so will happily take a dump when it comes down to it.
Re:“[Twitter] is doing 1000 poorly batched RPCs just to rend...
by karp773 on Sunday November 13, 2022 @04:03PM
(#33587322)
Not to defend Musk but, as a matter of fact, he has written his fair share of code so he probably knows what he's talking about in this case.
I just refreshed twitter's home page and for me it made 111 http requests (per Chrome's dev tools). That includes static resources like images and style sheets. So only ~10% are made from the browser and the other 90% must be due to microservice architecture on the backend, I suppose?
Re:“[Twitter] is doing 1000 poorly batched RPCs just to rend...
by paulryanrogers on Sunday November 13, 2022 @04:17PM
(#33587424)
Where can I read more about Musk's coding prowess?
Because the little I've gleaned is that he insisted on porting PayPal to Microsoft stack, got pushed out, and picked up enough about rocketry to impress Carmack in casual conversation.
"Elon Musk was appointed the Chief Technology Officer..."
He created a web based company as early as 1995. In a three person startup one cannot become a CTO without writing code. So my bet is he understands something about http vs local calls.
Re:“[Twitter] is doing 1000 poorly batched RPCs just to rend...
by kuberlog on Tuesday November 15, 2022 @11:59PM
(#33619082)
No, he was CTO of a 3 person startup. He was the ONLY one that knew how to code for a while. He did it pretty obsessively and slept next to his computer.
Re:“[Twitter] is doing 1000 poorly batched RPCs just to rend...
by Spivak on Sunday November 13, 2022 @03:25PM
(#33586967)
Does this account for pipelining and multiplexing because that’s like the whole reason those exist so that every client doesn’t have to think about batching and the browser can do it for you and present a fiction that they aren’t. Because Twitter definitely supports HTTP 1.1 and HTTP/2.
We’re currently in the process of ripping out a lot of client side opportunistic model hydration in favor of just making multiple requests because it’s faster.
Re:“[Twitter] is doing 1000 poorly batched RPCs just to rend...
by csande17 on Sunday November 13, 2022 @07:38PM
(#33588927)
Pipelining and multiplexing can certainly help make multiple requests faster, but they aren't a magic bullet. In particular, if your frontend makes a lot of requests sequentially (which is pretty common in badly designed frontends -- a component makes a request, waits for the response before rendering children, a child makes a request, etc...), that's still going to perform really badly in high-latency environments.
Of course, it's equally possible to mess up your server-side logic and make it really slow. And a lot of teams have really low performance expectations, or expect all of their customers to have 5G internet connections and live next to a data center, and either of those can buy you time to make requests.
Re:“[Twitter] is doing 1000 poorly batched RPCs just to rend...
by prirun on Sunday November 13, 2022 @03:33PM
(#33587044)
I'm guessing Twitter is a highly distributed system, so of course it's doing a bunch of poorly batched RPCs. That's more or less the definition of a highly distributed system, where all the data can't be stored in 1 place.
Re:“[Twitter] is doing 1000 poorly batched RPCs just to rend...
by nitwit005 on Sunday November 13, 2022 @09:55PM
(#33589754)
It can only be batched if the source and destination are the same. Different parts of the page likely hit different servers, and query different backends.
Re:“[Twitter] is doing 1000 poorly batched RPCs just to rend...
by nitwit005 on Monday November 14, 2022 @02:10PM
(#33599242)
I phrased it poorly, but you have an http2 connection to some load balancer, and it fans out different URL paths to different services. The RPC calls are presumably done between backend services.
The engineers brought in are in the position where they are expected to trash the existing teams code. And then in the context of a slow page load, they might look at it and see up to 1000 of something in RPC and tell Musk and it comes out in a slightly mixed up way when he tries to repeat it.
The reason the number of calls gets high is (I guess) not from 1000 individual different types of requests, but from pulling many pages or records each of several things.
Anyway it is easy to trash other people's code especially if that's what your boss wants you to do to justify bringing you aboard. I have been the victim of it and also on one or two occasions done the same thing myself to some degree.
Another aspect is that no engineer really wants to maintain someone else's code if they don't have to. They are going to have their own viewpoint on architecture etc. and coming onboard is a prime opportunity to start a rewrite. So slamming the existing architecture is just an easy way to try to get that authorized.
I think it's a bit of an integrity test coming in and talking about someone else's code base. But people are unfortunately creatures of circumstance, and these things can bring out the worst in some as far as fairness goes. But again, even for a fair person it might be easy in that position to scoff at a codebase.
Most people are not particularly high integrity though so will happily take a dump when it comes down to it.
I just refreshed twitter's home page and for me it made 111 http requests (per Chrome's dev tools). That includes static resources like images and style sheets. So only ~10% are made from the browser and the other 90% must be due to microservice architecture on the backend, I suppose?
Because the little I've gleaned is that he insisted on porting PayPal to Microsoft stack, got pushed out, and picked up enough about rocketry to impress Carmack in casual conversation.
"Elon Musk was appointed the Chief Technology Officer..."
He created a web based company as early as 1995. In a three person startup one cannot become a CTO without writing code. So my bet is he understands something about http vs local calls.
We’re currently in the process of ripping out a lot of client side opportunistic model hydration in favor of just making multiple requests because it’s faster.
Of course, it's equally possible to mess up your server-side logic and make it really slow. And a lot of teams have really low performance expectations, or expect all of their customers to have 5G internet connections and live next to a data center, and either of those can buy you time to make requests.
Also doesn’t the new fangled http/2 protocols and the like solve this, at least partially?
Why can't twitter have 1000?