News for hackers. Stuff that matters.
Re:The Evolution of SRE at Google by FuriouslyAdrift on Friday January 03, 2025 @04:00PM (#42589421)
I think the single biggest thing about Google SREs (at least in the early years) was that if your team was going to launch a new product, you had to have an SRE to help and to maintain the service.

Google deliberately limited the amount of SREs, so you had to prove your stuff worked and sell it to the SRE to even get a chance to launch.

Constraints help to make good ideas better...

Re:The Evolution of SRE at Google by hollowsunsets on Friday January 03, 2025 @04:32PM (#42589712)
It's not good when you have an SRE on hand to act as a babysitter of sorts. That is how some companies use SREs these days. They do the toil and sysadmin work so the product engineers can focus on features. Exactly what we hoped to avoid, but here we are.
Re:The Evolution of SRE at Google by sgarland on Friday January 03, 2025 @06:56PM (#42590811)
If by some you mean nearly all, then yes, and yes, it’s terrible.

Super fun being the adult in the room having to explain for the millionth time why someone can’t expect that a network call will always succeed, and will never experience latency.

Re:The Evolution of SRE at Google by nvarsj on Saturday January 04, 2025 @06:18AM (#42594063)
Even at Google it's like this. I spent the holidays watching my on-call Google SRE friend trying to diagnose misbehaving mapreduce jobs devs had written. They are basically glorified first line support so SWEs don't get woken up in the night.

Which seems like the worst possible setup to me - devs should be first on call for code they write. That seems like a basic principle to me and creates the correct incentives.

Re:The Evolution of SRE at Google by arthurjj on Friday January 03, 2025 @04:33PM (#42589723)
Thanks for this detail, I worked at Google, with SREs, and didn't know it. It seems like the type of 'design' detail that might be more important than this entire article
Re:The Evolution of SRE at Google by emtel on Friday January 03, 2025 @04:38PM (#42589756)
This culture was, imo, directly responsible for google's failure to launch a facebook competitor early enough for it to matter.

The Orkut project was basically banned from being launched or marketed as an official google product because it was deemed "not production ready" by SRE. Despite that it gained huge market share in Brazil and a few other countries before eventually losing to FB. By the time their "production ready" product (G+) launched it was hilariously late.

Facebook probably would have won anyway, but who knows what might have happened if Google had actually leaned into this very successful project instead of treating it like an unwanted step-child.

Re:The Evolution of SRE at Google by mike_hearn on Saturday January 04, 2025 @05:06AM (#42593811)
How was it banned from being launched? It did launch and the desire to not be promoted as a Google product came from Orkut himself, iirc.

The reason it was not regarded as 'production ready' was that the architecture didn't scale. In fact it also didn't run on the regular Google infrastructure that everything else used and that SRE teams were familiar with; it was a .NET app that used MS SQL Server.

This design wasn't a big surprise. Facebook won not because Orkut lost but because Facebook were the first to manage gradual scaleup without killing their own adoption, by figuring out naturally isolated social networks they could restrict signup to (American universities). This made their sharding problem much easier. Every other competitor tried to let the whole world sign up simultaneously whilst also offering sophisticated data model features like the ability to make arbitrary friendships and groups, which was hard to implement with the RDBMS tech of the time.

Orkut did indeed suffer drastic scaling problems and eventually had to be rewritten on top of the regular Google infrastructure, but that just replaced one set of problems with another.

Re:The Evolution of SRE at Google by ghaff on Saturday January 04, 2025 @09:17AM (#42594782)
Of course restricting rollout to American university emails (including alumni addresses--at least at one point) was also a pretty natural consequence of Facebook's origins.
Re:The Evolution of SRE at Google by emtel on Saturday January 04, 2025 @12:23PM (#42596120)
The attitude within SRE toward Orkut (the product) was one of disdain if not contempt. A healthy culture does not treat rapidly growing products this way.
Re:The Evolution of SRE at Google by mike_hearn on Saturday January 04, 2025 @02:38PM (#42597096)
I mean, I'm personal friends with a former Orkut SRE. The idea that Google SRE ignored or disdained Orkut just isn't right. Nonetheless, if your job is defined as "make the site reliable" and it's written in a way that can never be reliable then it's understandable that you're going to have at least some frustrations.
Re:The Evolution of SRE at Google by emtel on Tuesday January 07, 2025 @11:47PM (#42631027)
Search SRE were huge dicks about it. I felt bad for the Orkut SREs.
Re:The Evolution of SRE at Google by philll on Saturday January 04, 2025 @01:34PM (#42596659)
That sounds dysfunctional.