Re:Why Twitter didn’t go down: From a real Twitter SRE
by darkwater on Tuesday November 22, 2022 @08:19AM
(#33705421)
> Our biggest site (about dozen million users, a bunch of services and caching underneath, few gbits of traffic) took zero actual maintenance for 2022, "it just works", any job was implementing new stuff. It took some time to get to that state but once you do aside from hardware failures it "runs itself"
Nobody is adding changes that blows out the DB? or add some inefficient code that burns CPU much faster?
Re:Why Twitter didn’t go down: From a real Twitter SRE
by chasd00 on Tuesday November 22, 2022 @01:09PM
(#33708822)
> Nobody is adding changes that blows out the DB? or add some inefficient code that burns CPU much faster?
not to be flippant but you find that out like 3 environments below production.
You might have 200 different apps (hell, we have close to that, only 3 people in ops) but competent team will make sure they deploy in same way and are monitored in same way.
And once you go from "a server" to multiple servers, whether the end number ends up being 20 or 200 isn't that important till you start hitting say switching capacity, and if you're in cloud that's usually not your concern anyway.
Our biggest site (about dozen million users, a bunch of services and caching underneath, few gbits of traffic) took zero actual maintenance for 2022, "it just works", any job was implementing new stuff. It took some time to get to that state but once you do aside from hardware failures it "runs itself"