News for hackers. Stuff that matters.
Re:The Evolution of SRE at Google by rzz3 on Friday January 03, 2025 @01:40PM (#42588298)
These days I wonder if Google is really the example to follow. There was a time 10 or 15 years ago where Google seemed to be leading the industry in everything, and I feel like a lot of people still think they do when it comes to engineering culture. These days I tend to see Google as a bit of a red flag on a resume, and I have a set of questions I ask to make sure they didn’t drink too much of the koolaid. Perhaps more importantly, when I look at Google from the outside these days, I see that their products have really gone downhill in terms of quality. I see Google Search riddled with spam, I see Gemini struggling to keep up with OpenAI, Google Chat trying to keep up with Slack but missing the mark, Nest being stagnant, I could go on and on. All this to say that I don’t think Google is the North Star that it used to be in terms of guiding engineering culture throughout the industry.
Re:The Evolution of SRE at Google by scarface_74 on Friday January 03, 2025 @02:29PM (#42588701)
I would never hire a product person from Google or someone I needed to be visionary. For the most part, their products suck, they have no vision and no follow through.

But their technology is top notch. I hire mostly for startups and green field initiatives though and I wouldn’t hire anyone from any BigTech company unless I had “hard” technical problems to solve.

Yes I’ve done a stint at BigTech.

Re:The Evolution of SRE at Google by ninkendo on Friday January 03, 2025 @02:44PM (#42588811)
They have top notch tech, yes, but it’s massively overkill for literally every company that’s not at google’s scale. If you’re not careful you may hire someone who will try to replicate everything google does, when you may need only 1% of the complexity. This is the experience I’ve generally had with xooglers… they lament that they don’t have the same tools/tech stack they had at google, and so their first act is to try to move everything to the closest open source equivalents, even if they’re not a good fit.

There’s good things and bad things to take away from experience at google… you have to be careful to ignore things that won’t actually help you.

Re:The Evolution of SRE at Google by scarface_74 on Friday January 03, 2025 @03:46PM (#42589318)
I agree. I haven’t run into a “hard problem” in my career

By hard problem I mean technically at the top 5% of a problems in the industry that can’t be solved by throwing money at a SaaS or using a cloud provider.

Re:The Evolution of SRE at Google by everfrustrated on Friday January 03, 2025 @05:57PM (#42590374)
Most companies are really just crud apps. Very few are doing anything technically innovative. And that's just fine.

I wonder how much of the early Google technical innovation was more a product of open source tech/distributed systems being a lot more immature (I'm particularly thinking databases) 25 years ago.

Ultimately all companies get bloated and loose their way. It shouldn't be a suprise this has happened to Google - 25 years on they are mega corp and idling. Probably for the best as it allows innovators a chance to compete.

Re:The Evolution of SRE at Google by deepsun on Friday January 03, 2025 @05:00PM (#42589882)
I've been the "you're not google" person for several years, but now softened my position.

The thing is -- it depends. Sometimes when everyone knows some complex system well -- it becomes easy.

One example comes to mind -- Kubernetes. 90% of teams don't need all its complexity. And I've been "you don't need it" person for some time. But now I see that when everyone knows it -- it's actually much easier to deploy even simple websites on it, because it's a common lingo and you don't spend time explaining how it's deployed.

It's not like civic engineers, when an over-engineered bridge would cost a lot more in materials.

Re:The Evolution of SRE at Google by scarface_74 on Friday January 03, 2025 @05:17PM (#42590052)
If you have a simple website , you can containerize your backend and use much simpler services from AWS and serve your static assets on S3.

Kubernetes is rarely the right answer for simple things even if Docker is.

Re:The Evolution of SRE at Google by deepsun on Friday January 03, 2025 @06:43PM (#42590728)
Your response is the perfect example of my point. Each time you use "much simpler services" you still _need to explain_ the setup for the simpler services. Someone might know it, someone not. E.g. some project may eventually grow out of Lambda RAM limitations, but noone in the team knew that. While Kubernetes is one-size-fits-all setup, even if I don't like it.

And yes, I use the Cloud Run myself, but only for my one-person projects. For the team projects consistency is much more important (same way to access/monitor/version etc).

PS: I would say even AWS/GCP is already a huge overkill for most projects. But for some reason you didn't see exactly the same problem starting with clouds right away.

Re:The Evolution of SRE at Google by scarface_74 on Friday January 03, 2025 @07:56PM (#42591182)
Lambda can use up to 10GB of RAM and there is also App Runner.

And “using AWS” can be as simple as starting off with Lightsail - a VPS service with fixed pricing

https://aws.amazon.com/lightsail/

Re:The Evolution of SRE at Google by deepsun on Saturday January 04, 2025 @01:51AM (#42593009)
RAM is just one example. Every simpler service has its limitations, and if everyone (including new hires) knows the simpler service well -- it's perfect. E.g. in my experience everyone knew App Engine at some point and it worked well for us. Now it's a zoo of devops pieces, so I tolerate Kubernetes only because everyone kinda knows it.

And the Kubernetes was just one example of my "you're not google" point. There is many more technologies that are definitely overkill, but is a good common denominator, even when it's 1000x more complex than needed for the task at hand.

PS: Btw, I dunno why people downvoted your comment. It's fits the HN "Guidelines" at the bottom, so upvoted.

Re:The Evolution of SRE at Google by mschuster91 on Friday January 03, 2025 @05:38PM (#42590239)
> much simpler services from AWS

Like what, Lambda? I've seen so much horrible hacks and shit done with it (and other AWS services cough API gateway cough), these days I rather prefer a set of Kubernetes descriptors and Dockerfiles.

At least that combination all but enforces people doing Infrastructure-as-a-code and there's (almost) no possibility at all for "had to do live hack XYZ in the console and forgot to document it or apply it back in Terraform" .

Re:The Evolution of SRE at Google by icedchai on Friday January 03, 2025 @05:51PM (#42590335)
In my experience, you are better off with ECS/Fargate than Lambda for serving an API. You get much more flexibility.

Also, I've witnessed people editing Lambda code through the console instead of doing a real deploy. what a mess...

Re:The Evolution of SRE at Google by scarface_74 on Friday January 03, 2025 @06:10PM (#42590465)
You can’t edit Lambda code in the console when you deploy a Docker image to Lambda.

As far as flexibility, while there have been third party libraries that let you deploy your standard Node/Express, .Net/ASP, Python/Flask app to Lambda, now there is an official first party solution

https://github.com/awslabs/aws-lambda-web-adapter

And as far as ECS, it is stupid simple

I’ve used this for years

https://github.com/1Strategy/fargate-cloudformation-example/...

Re:The Evolution of SRE at Google by mschuster91 on Friday January 03, 2025 @06:50PM (#42590780)
> Also, I've witnessed people editing Lambda code through the console instead of doing a real deploy. what a mess...

Yeah, exactly that's what I am talking about. Utter nightmare to recover from, especially if whoever set it up thought they needed to follow some uber-complex AWS blog post with >10 involved AWS services and didn't document any of it.

Re:The Evolution of SRE at Google by scarface_74 on Friday January 03, 2025 @07:57PM (#42591188)
You can’t edit Lambda code directly in the console when using Docker deployments
Re:The Evolution of SRE at Google by scarface_74 on Friday January 03, 2025 @05:50PM (#42590328)
AWS App Runner

https://aws.amazon.com/blogs/containers/introducing-aws-app-...

Google has something similar.

Re:The Evolution of SRE at Google by icedchai on Friday January 03, 2025 @05:55PM (#42590366)
GCP has Cloud Run, which looks similar. App Runner is basically a wrapper on top of Fargate, right?
Re:The Evolution of SRE at Google by scarface_74 on Friday January 03, 2025 @06:13PM (#42590498)
Yep. Every “serverless” compute service is just a wrapper on top of Firecracker including Lambda and Fargate.
Re:The Evolution of SRE at Google by elktown on Saturday January 04, 2025 @06:36AM (#42594116)
The problem is that it can create a chain-reaction of complexity because it opens up possibilities for over-engineering. In the sense of: "Yes, it's a bit over-engineered, but k8s makes it manageable for us anyway!" - consciously or subconsciously. When I'd often suspect that some restrictions in what's possible/acceptable would've created a significantly leaner overall design in the end.
Re:The Evolution of SRE at Google by VirusNewbie on Saturday January 04, 2025 @01:48AM (#42592998)
> but it’s massively overkill for literally every company that’s not at google’s scale.

Before I worked at Google I was at a small telecom company that was running into limits of what some of the Dataflow/Apache Beam product could do, so we had to rewrite it (and commit it back to Beam).

There are companies that have massive scaling issues even if they're not planet scale cloud providers or something.

You can replicate a lot of Google tech now by....just using the OSS they release and/or jumping on a modern cloud provider (GCP or AWS). It's not 2012. You can use a good database and not have to reinvent it.

Re:The Evolution of SRE at Google by jarsin on Friday January 03, 2025 @04:08PM (#42589478)
And now we are all stuck doing leetcode interviews primarily because of Google.
Re:The Evolution of SRE at Google by marssaxman on Friday January 03, 2025 @05:00PM (#42589888)
Leetcode didn't exist back then; the site was founded a little less than a decade ago.
Re:The Evolution of SRE at Google by jarsin on Friday January 03, 2025 @07:02PM (#42590839)
I was using "Leetcode" as in style of the interview. The Leetcode website was founded due to everyone in the industry copying google/big tech in these style of interviews.
Re:The Evolution of SRE at Google by mike_hearn on Saturday January 04, 2025 @06:37AM (#42594121)
Google copied that idea from Microsoft, primarily.
Re:The Evolution of SRE at Google by fweimer on Saturday January 04, 2025 @05:47AM (#42593932)
The flip side from this assumption about their technology is that if some service is not working, people are very quick to blame the impacted (paying) user. “You are running into rate limits”, “Google is applying anti-abuse controls to your account”, and so on. But at least for some services, I strongly suspect we are actually experiencing random system failures. In my experience, it's rare to get acknowledgement of this from Google. Tickets may not even make it to them because of this pervasive “Google technology is perfect” assumption. The end result feels a bit like gaslighting (doubting our sanity because we can't spot the pattern that is supposed to be obvious): we are encouraged to attribute meaning to more or less random reactions from a complex system.
Re:The Evolution of SRE at Google by dehrmann on Friday January 03, 2025 @01:43PM (#42588323)
I agree on the product and customer service front, but Google's reliability is top-notch.
Re:The Evolution of SRE at Google by mirashii on Friday January 03, 2025 @02:22PM (#42588649)
As a Google Cloud customer, I’d say it might be best to split Google into some divisions or something, as Google Cloud’s reliability is a relative shitshow compared to Google.com.
Re:The Evolution of SRE at Google by brudgers on Friday January 03, 2025 @03:54PM (#42589376)
Unless you have Google sized problems and resources, Google probably is not the best example because the things Google does are done to address Google size problems with Google sized resources. It's tooling and methods are not commercial products.

For example, Google can get away with the flaws of it's AI search results because it is Google.

Re:The Evolution of SRE at Google by jeffbee on Friday January 03, 2025 @04:11PM (#42589502)
The fact that some people prefer ChatGPT over Gemini is not something that SRE can help you with. The fact that ChatGPT is rarely available is something that SRE could help Microsoft avoid.
Re:The Evolution of SRE at Google by lupire on Friday January 03, 2025 @04:32PM (#42589709)
ChatGPT is rarely available??
Re:The Evolution of SRE at Google by jeffbee on Friday January 03, 2025 @04:40PM (#42589768)
They have major, long-lasting incidents at least once a week. https://status.openai.com/
Re:The Evolution of SRE at Google by dilyevsky on Saturday January 04, 2025 @07:30PM (#42598717)
Not even three nines so…
Re:The Evolution of SRE at Google by taeric on Friday January 03, 2025 @01:44PM (#42588332)
I'm curious what you have in mind for evidence of "koolaid" there?

Hard not to disagree with the general trend you are outlining. Most of that feels driven by product choices, moreso than execution. I think a lot of the previous glorification of their work was likely misguided, as well. But I would be hard pressed to be quantitative on that.

Re:The Evolution of SRE at Google by scarface_74 on Friday January 03, 2025 @02:30PM (#42588708)
It was 5 years after Android was introduced that the CEO stopped using BlackBerry…
Re:The Evolution of SRE at Google by taeric on Friday January 03, 2025 @03:59PM (#42589414)
An amusingly good quantification of some evidence. Well done! :D

Still, I don't have much to say that I think the engineering was overly good or bad. I typically think that what they captured for a short while, at least, was enthusiasm. In particular, developers were enthusiastic to be near Google technology in a way that I don't think I've seen for other companies, since.

I don't think they identified it as such, though. Which could be why they seem slow to see that a lot of that has evaporated.

Not to say that they have no enthusiasm, now. I'd wager they still have a lot. But as a percentage share of all developers, it feels very different.

Re:The Evolution of SRE at Google by jofla_net on Friday January 03, 2025 @01:44PM (#42588327)
Re:The Evolution of SRE at Google by sanj on Friday January 03, 2025 @02:21PM (#42588645)
Re:The Evolution of SRE at Google by znpy on Friday January 03, 2025 @03:15PM (#42589073)
it shouldn't have happened in the first place.
Re:The Evolution of SRE at Google by pphysch on Friday January 03, 2025 @01:43PM (#42588326)
Who is then?
Re:The Evolution of SRE at Google by scarface_74 on Friday January 03, 2025 @02:34PM (#42588738)
From a product standpoint every BigTech company has done better at releasing new products than Google.
Re:The Evolution of SRE at Google by pphysch on Friday January 03, 2025 @02:56PM (#42588913)
How did we get from SRE culture to (paraphrasing) "I personally think Google makes worse products than IBM, Oracle, Apple, Netflix, Broadcom, et al."
Re:The Evolution of SRE at Google by scarface_74 on Friday January 03, 2025 @03:43PM (#42589297)
Having good technology and good products are orthogonal. People are conflating the two
Re:The Evolution of SRE at Google by hollowsunsets on Friday January 03, 2025 @04:33PM (#42589729)
What defines a good product? Something that many customers use? Something that makes shareholders happy?
Re:The Evolution of SRE at Google by scarface_74 on Friday January 03, 2025 @05:13PM (#42590005)
A product that either moves the needle as far as revenue and/or makes the ecosystem better. It also needs to be a product that gets continuously better as long as there is a market for it and not abandoned quickly.

- “a connected TV device”. How many cancelled lines of products have they abandoned? How many market failures have they had in their own line of phones? The Pixel’s aren’t taking the world by storm and they spent billions on Motorola and then sold it off for scraps

They have been releasing a cancelling their own tablet initiatives for years.

At one point they had 5 separate messaging initiatives going on simultaneously.

Even today they have three operating system initiatives that are not on the same codebase - Fuscia, Android and ChromeOS.

They have basically abandoned Flutter and don’t use it for any of their high profile apps.

What have they actually done besides ads?

And the obvious evidence is their money losing “other bets”

Also Google Fiber

https://www.spglobal.com/marketintelligence/en/news-insights...

Re:The Evolution of SRE at Google by yodsanklai on Friday January 03, 2025 @04:31PM (#42589699)
> There was a time 10 or 15 years ago where Google seemed to be leading the industry in everything

They used to write interesting books and articles about software engineering. It felt that they were maintaining high quality standards and were an industry reference. Nowadays, I wouldn't go as far as saying it's a red flag to have Google on one's resume, but definitely not the same appeal as before.

Re:The Evolution of SRE at Google by nimish on Friday January 03, 2025 @07:10PM (#42590886)
Goodhart's Law for employment