News for hackers. Stuff that matters.
Re:Kubernetes: The Documentary by tester756 on Sunday February 13, 2022 @01:44PM (#30324331)
Why Google despite having Borg and know-how so early wasn't 1st to the Cloud? or bigger threat to AWS/Azure?
Re:Kubernetes: The Documentary by ec109685 on Sunday February 13, 2022 @04:02PM (#30325660)
Borg was good enough that google engineers didn’t have any desire to use GCP and thus most of the company was not dogfooding its own product.

Additionally, the “two pizza teams” style of Amazon’s development was more amenable to the AWS model of development where each piece of Amazon could run effectively in AWS [1]. Google is structured differently, and it’s only recently that both AWS and GCP could even approach running the largest systems from a company like Google.

[1] Yes I know that AWS wasn’t built specifically for Amazon engineers, but the fact that it could accommodate them over time has been a boon for their cloud, and it gives enterprises confidence that of AWS’s strategic importance and desire to make it world class. There isn’t a large hidden layer of services that Amazon has access to that the general public doesn’t.

Re:Kubernetes: The Documentary by deanCommie on Sunday February 13, 2022 @03:10PM (#30325130)
Hubris.

Steve Yegge talks about it on his podcast: https://youtu.be/9v4z46Ea35Q

Amazon actually works backwards like they say - they ask customers what they want, and build it.

Google thinks they're smarter than everyone else so they can't comprehend that developers out there in the world want different things than the developers inside Google.

You can see this permeate even in this documentary, which is ostensibly about a success - when Google actually did launch something that developers really liked - K8S.

They didn't start from developer needs or wants. They started by noticing AWS was eating their lunch, and wanting to compete with them, but realizing they couldn't win by doing it directly.

Re:Kubernetes: The Documentary by ec109685 on Sunday February 13, 2022 @03:33PM (#30325358)
To be fair, K8s has caught on even outside of GCP.

Also, while Amazon does work backwards from customers, part of them working backwards is how to translate customer requirements into a successful business. Amazon is doing this as a for profit company, not spending money on altruistic initiatives.

E.g. purely for the benefit of customers, having AWS services be multi-cloud and open source or reducing egress cost would be best.

Re:Kubernetes: The Documentary by p_l on Sunday February 13, 2022 @06:17PM (#30326590)
Because K8s, while commonly deployed on the cloud, is more of a "local cloud operating system" that you use to marshall a bunch of resources into "cloud" (and finalizing TPRs into CRDs really made it shine).

To the point that k8s is base operating system for... Chick-Fil-A restaurants, deployed locally, controlling the industrial kitchen hardware.

Re:Kubernetes: The Documentary by singron on Sunday February 13, 2022 @03:04PM (#30325067)
Also, Google's infrastructure is so vertically integrated that it's not obvious how to offer it others. E.g. Borg assigns 1 IP to each machine, which means every pod-equivalent has to listen on a dynamically allocated port, so Google eschewed DNS in favor of a system that more easily assigned dynamic IP:port pairs without relying on TTLs (BNS/Chubby). Google also decoupled storage from compute and had a bunch of custom databases to run in that paradigm (GFS/CFS/BigTable).
Re:Kubernetes: The Documentary by ra7 on Sunday February 13, 2022 @03:26PM (#30325285)
I’ve heard Google considers (considered?) their infrastructure prowess as a secret sauce. Could it be that they didn’t want anyone else to benefit from it?

They had years of headstart on most infrastructure technology that’s ubiquitous now, but didn’t bother to offer it to others. Perhaps they thought only Google should benefit from it, while Amazon saw a business opportunity with AWS.

Re:Kubernetes: The Documentary by oblio on Sunday February 13, 2022 @03:33PM (#30325367)
I'm also curious about this.

The empirical evidence seems to point that way, a lot of people were noticing that many of the Google stuff became open source or got open source equivalents from Google (Borg/Kubernetes, Blaze/Bazel, etc.) only after a critical mass of former Google employees left Google and since they liked Google tooling started replicating it as open source software outside of Google. In some cases that open source software got traction and Google seemed to be falling behind in some areas.

Which basically tells me that in the software world, you can't really have trade secrets. I could be wrong, though.

Re:Kubernetes: The Documentary by smarterclayton on Sunday February 13, 2022 @06:57PM (#30326897)
One of the stories I provided for the documentary that didn’t make it (but is one of my favorites) relates to an early 2015 discussion I had with Brian Grant (googler / architect / worked on Omega / also in the documentary).

Brian writes very succinct and clear descriptions of problems encountered inside google, breaks them down to “what worked, what didn’t work”, and then would summarize a design seed in a github comment (of which several such subsystems subsequently were implemented that have been fairly successful).

I would keep mental track of some of these comments and assign a score based on how much R&D investment at google was compressed into a 200-300 word comment. I guesstimated some at being valued at upwards of $100k a word, and jokingly asked brian if he was authorized to share $10M worth of google’s secret sauce. He laughed, but reminded me of some of the same things mentioned in the documentary (which we all knew in the project at the time) - paraphrased:

“if you can seed a community project that includes some of these patterns built on that research, and that enables $10 billion of application team value, and google can capture $1B of that in google cloud, that’s a pretty good return on investment.”

And definitely among the key contributors I worked with from google was the very human and very relatable (for HN) desire to build that shows those internal ideas off and “do a successor to borg that really takes what we’ve learned and doesn’t just copy it”.

Even if it was a business strategy to help catch google cloud up… the beauty of open source is that you can benefit from it regardless, and it moves the industry forward in a meaningful way.

Re:Kubernetes: The Documentary by potamic on Monday February 14, 2022 @03:35AM (#30329670)
> Brian writes very succinct and clear descriptions of problems encountered inside google, breaks them down to “what worked, what didn’t work”, and then would summarize a design seed in a github comment (of which several such subsystems subsequently were implemented that have been fairly successful). I would keep mental track of some of these comments and assign a score based on how much R&D investment at google was compressed into a 200-300 word comment

Most places I have seen, architects seem to spend most of their time writing out overly verbose documents, detailed specifications, making presentations etc etc. There is pressure to show you are in charge of the project and accountable for it so you end up with all these ceremonies where you create heaps of content and dump it onto the teams.

Whereas the sort of situation you described is really where the role of an architect shines, in making sense of the chaos and showing light to the teams. You often don't need more than a short message or a brief meeting to convey these ideas, but such things don't lend upward visibility to what you're doing. I've always wondered if the life of an architect in Google involves these things or is quite different from most organizations?

Re:Kubernetes: The Documentary by disgruntledphd2 on Sunday February 13, 2022 @05:51PM (#30326401)
If you look at development of lots of "Big Data" stuff (Hadoop is a good example), Google shipped papers, while FB shipped code (not entirely true for Hadoop, but much more true for some of Google's later papers).
Re:Kubernetes: The Documentary by thockingoog on Monday February 14, 2022 @12:26AM (#30328669)
This was, literally, one of the arguments for building and releasing kubernetes. The rise of Hadoop made it much harder to justify MapReduce being different.

If we just talked about Borg, but didn't ship code, someone else might have set the agenda, rather than Kube.

Re:Kubernetes: The Documentary by buck4roo on Sunday February 13, 2022 @03:54PM (#30325579)
Don't forget Prometheus (clone of borgmon by ex-SRE)
Re:Kubernetes: The Documentary by jeffbee on Sunday February 13, 2022 @04:01PM (#30325654)
There is no way to perceive Prometheus as Google falling behind, though. It's more of a lesson about the perverse willingness of the public to subject themselves to terrible technologies that Google began deprecating years ago.
Re:Kubernetes: The Documentary by oblio on Sunday February 13, 2022 @04:18PM (#30325766)
Why is Prometheus terrible? It's pretty cool.
Re:Kubernetes: The Documentary by jeffbee on Sunday February 13, 2022 @04:23PM (#30325819)
At small scale it's fine. Its architecture totally forecloses any possibility of distributed evaluation, so there is a brick wall of scalability. Its human-readable-text-over-HTTP scraping scheme is also hilariously inefficient. It is not, in short, "planet scale".

https://www.vldb.org/pvldb/vol13/p3181-adams.pdf

Re:Kubernetes: The Documentary by oblio on Sunday February 13, 2022 @05:19PM (#30326203)
Well, I get the point of Google not falling behind with their tech, but even most Google competitors aren't working at Google scale.

For Prometheus they could have just chosen to aim for smaller businesses, since that's 99% of businesses out here.

So it could just be a tradeoff.

Re:Kubernetes: The Documentary by jeffbee on Sunday February 13, 2022 @06:05PM (#30326494)
Well, to be fair, they are offering Prometheus as a service, and they built it as a façade upon Monarch, which pretty convincingly demonstrates the superiority of Monarch.

I just find a kind of perverse humor in the fact that outside google Prometheus is viewed as alien technology - it even has that condescending name - but inside Google borgmon was considered the punch line of memes and/or a hazing ritual.

Re:Kubernetes: The Documentary by p_l on Sunday February 13, 2022 @06:14PM (#30326565)
IIRC, a core point in early days of Prometheus was a greatly simplified system that built on some of the same ideas, but without the parts that made it into punch line of memes or a hazing ritual for new guys.

While the text based metrics aren't best, they are a reasonable compromise, and I found that it provides a good enough common interface where if needed I can change what collects from it later on.

Are there better options? SURE! I can, however, usually get my 80% of the problem solved with Prometheus integration and worry about scaling later.

Re:Kubernetes: The Documentary by daveevad on Monday February 14, 2022 @12:21AM (#30328651)
human-readable-text-over-HTTP is planet scalable
Re:Kubernetes: The Documentary by samstave on Sunday February 13, 2022 @05:52PM (#30326410)
Google and FB (and apple, but less so?) -- have been making their own servers and hardware for years...

They protect this aggressively, and the tech is impressive.

I remember the first time I accidentally saw a custom google node when I went there to pick a friend up from there, and I had gone to his desk and I saw a really nice looking PCB on his desk he forgot about, and I exclaimed "WHATs that' -- I was pretty familiar with a wide range of server boards, and I knew that was custom immediately...

He panicked covered it and we didnt really talk about it much, but it was known really early that they were building a lot of custom kit.

~2005 maybe?

Re:Kubernetes: The Documentary by deanCommie on Sunday February 13, 2022 @03:13PM (#30325159)
The mistake also reveals the constrained imagination: Google could only conceive of offering what they had internally to others, struggled, and continues to struggle.

Amazon didn't try - they just knew that they were good at building high scale distributed systems, and thought they could build services for others. That's what S3 and SQS and EC2 were - built from scratch, not trying to figure out how to externalize the internals.

Re:Kubernetes: The Documentary by ec109685 on Sunday February 13, 2022 @04:19PM (#30325779)
GCP was built from scratch as well.
Re:Kubernetes: The Documentary by singron on Monday February 14, 2022 @08:06PM (#30341151)
It's more complicated, but I think everything is at least a substantial layer on top of an internal offering. E.g. GCE runs VMs in Borg using the same physical infrastructure used for other services, but GCE has its own API and virtualized networking on top that hides Borg and was not (at the time) used for any internal services.
Re:Kubernetes: The Documentary by jthrowsitaway on Sunday February 13, 2022 @01:48PM (#30324362)
Google's advertising business is really profitable so anything else is secondary. Amazon came at it from a business angle. They had a ton of spare resources that they only needed once in a while and figured out a way to make use of them.
Re:Kubernetes: The Documentary by nevir on Sunday February 13, 2022 @02:50PM (#30324960)
> They had a ton of spare resources that they only needed once in a while and figured out a way to make use of them.

Not actually the case. AWS was a completely independent stack (from hardware on up) - we couldn't use it internally for quite some time (years); and even then, only in a limited fashion (e.g. only S3 for some time)

Re:Kubernetes: The Documentary by jeffbee on Sunday February 13, 2022 @04:05PM (#30325687)
Bingo. There are two facts you need to understand Google's relationship with GCP. Cloud margins are the worst margins of anything at Google, so if there has to be a choice on where to spend money, it can only go to Cloud begrudgingly, or as part of some long game. Second thing you need to know is Google's internal workloads are way, way bigger than you are imagining. So if you see a headline that Google invested $100 billion in datacenters, don't imagine that all, most, or even a lot of that went toward GCP capacity. They spend that money on search, advertising, and letting an AI play ten million years of StarCraft.
Re:Kubernetes: The Documentary by randomsilence on Sunday February 13, 2022 @02:07PM (#30324555)
>They had a ton of spare resources

Is this true? There was the story about the resources they need for Christmas shopping that went unused for the rest of the year.

But recently somebody mentioned here (I don't remember where) that AWS was a separate business and that Amazon itself initially didn't use any resources from AWS.

Re:Kubernetes: The Documentary by FranGro78 on Sunday February 13, 2022 @02:21PM (#30324695)
That’s correct, the multi year project was known internally as MAWS - Move to AWS.
Re:Kubernetes: The Documentary by dastbe on Sunday February 13, 2022 @02:48PM (#30324943)
building a public cloud is more or less a rebuild of any infrastructure you currently have. it would be just unwise for a company to build their private cloud with the same trade offs as public cloud.

so while there’s the ideal state of running amazon on a large multi tenant cloud, there was absolutely no safe and reasonable path to making that happen by extending the existing infra. it would’ve also hampered aws by forcing them to focus way too large way too early.

Re:Kubernetes: The Documentary by pid-1 on Sunday February 13, 2022 @02:26PM (#30324737)
amazon.com also was really profitable when AWS was launched. IDK, at this point I think we can just agree Google is bad at diversifying. The day their ad business dies, they're done.
Re:Kubernetes: The Documentary by emodendroket on Sunday February 13, 2022 @02:41PM (#30324876)
The fear that they were plateauing was a major motivator. Google doesn’t really face that.
Re:Kubernetes: The Documentary by kortilla on Sunday February 13, 2022 @02:29PM (#30324762)
It was profitable, but nothing like Google’s margins. It was still just a glorified Walmart with delivery (i.e. not much of a moat).
Re:Kubernetes: The Documentary by ec109685 on Sunday February 13, 2022 @04:22PM (#30325812)
Amazon.com wasn’t really profitable. They ran at low margins for years. In 2006, their profit was like 500m against 10B in revenue.
Re:Kubernetes: The Documentary by jethro_tell on Sunday February 13, 2022 @06:22PM (#30326622)
Because they were using all profit to build out EC2, fulfilment centers and fullfilment tech. It wasn't that there wasn't profit because the margins were low, there wasn't profit because they were pouring it all into different verticals and building moats.

So yeah they weren't profitable but they weren't just scraping by either. I think that's one of those things that people miss when they compare a not-so-profitable startup to Amazon. The size, scale, and huge number of product markets that Amazon plays in is really pretty astounding for a company that is only 25 years old.

Re:Kubernetes: The Documentary by oblio on Sunday February 13, 2022 @03:35PM (#30325380)
That could take a ton of time to die off and they could still pivot.

But they're the least diversified, after Facebook, agreed.

Re:Kubernetes: The Documentary by mohanmcgeek on Sunday February 13, 2022 @01:48PM (#30324364)
They didn't realise it was a high margin business until AWS numbers were reported separately.

A lot of people used to think that Amazon.com was subsidising AWS. Financials from other VPS providers suggested this to be the case.

Re:Kubernetes: The Documentary by ec109685 on Sunday February 13, 2022 @03:57PM (#30325610)
GCP has been in the cloud business since 2008, two years after aws launched. They clearly were in a position to predict AWS’s profit margins before AWS publicized them given they were competing for same customers.
Re:Kubernetes: The Documentary by jakupovic on Sunday February 13, 2022 @04:21PM (#30325803)
I remember when GCP launched, we had an all hands meeting for all of AWS, which all fit in a big conference room, less than 100 people. People thought now that Google got in the business that was it. The leadership, Andy Jassy, assured everyone that our jobs were safe and we could get back to work so to say. Seems funny now, but at the time AWS was S3 and EC2 and google was ahead it seemed.
Re:Kubernetes: The Documentary by mohanmcgeek on Sunday February 13, 2022 @11:33PM (#30328387)
When was this? 2012?
Re:Kubernetes: The Documentary by jakupovic on Monday February 14, 2022 @05:56AM (#30330552)
2008 when GCP launched.
Re:Kubernetes: The Documentary by mohanmcgeek on Sunday February 13, 2022 @11:32PM (#30328385)
Compute Engine was announced at Google I/O 2012. GA in 2013

Google app engine was definitely NOT competing with AWS. They were not selling to enterprises.

Re:Kubernetes: The Documentary by emodendroket on Sunday February 13, 2022 @02:44PM (#30324911)
One theory I’ve heard (I think from Yegge?) is that Google’s internal tooling is so good at dealing with breaking changes that they’re too cavalier about introducing them.
Re:Kubernetes: The Documentary by throwaway984393 on Sunday February 13, 2022 @03:04PM (#30325071)
Even if that were the case, they clearly don't want to invest any extra time or money in refactor. So much of Kubernetes' design has been problematic, yet they just limp on monkey-patching and bolting on features that take years to go from idea to implementation. If you want any major changes or simplification in K8s, wait 3 years.

A traditional FOSS project is not limited in this way. The developers often just start a new major version and overhaul everything. But they don't have to answer to "the business" or huge institutional users. Commercially driven open source will always be a product beholden to stakeholders, rather than technology designed to operate as well as it can.

Re:Kubernetes: The Documentary by oblio on Sunday February 13, 2022 @03:39PM (#30325421)
> A traditional FOSS project is not limited in this way. The developers often just start a new major version and overhaul everything.

And regular users hate this.

Jamie Zawinski describes this funnily, eloquently and also offensively:

https://www.jwz.org/doc/cadt.html

> This is, I think, the most common way for my bug reports to open source software projects to ever become closed. I report bugs; they go unread for a year, sometimes two; and then (surprise!) that module is rewritten from scratch -- and the new maintainer can't be bothered to check whether his new version has actually solved any of the known problems that existed in the previous version.

But he's right, and that's why:

* regular application developers who don't care about ideology

* and mass market users

hate desktop Linux.

Re:Kubernetes: The Documentary by throwaway984393 on Sunday February 13, 2022 @08:01PM (#30327317)
Oh sure, that's 100% true. Patches often are applied by packagers when the upstream won't apply it themselves. But there's a big difference between patches for bugs and poor design limping on because we can't change it because people depend on it.

Users just want stable software. Developers want software that works better and solves more problems. The compromise is for users to stick with old major versions. At some point that's no longer tenable, but at least this way the software is free. Certainly no worse than when the same thing happens but you're paying for it.

Re:Kubernetes: The Documentary by jasonjayr on Tuesday February 15, 2022 @01:45PM (#30350847)
(don't click links to jwz.org, he hates news.ycombinator.com, you have to copy/paste so the Referrer doesnt' reflect this site.)
Re:Kubernetes: The Documentary by emodendroket on Sunday February 13, 2022 @03:33PM (#30325369)
The last task I want to be doing is having to migrate my Kubernetes definition files to accommodate someone's bright new idea.
Re:Kubernetes: The Documentary by throwaway984393 on Sunday February 13, 2022 @07:58PM (#30327291)
You already have to do that every 9 months (or is it 12 now?) as there's no "support" after that. But the point behind major versions is you don't change your definition files as long as you stay on the old major version, which is supposed to be supported for a long time.
Re:Kubernetes: The Documentary by odonnellryan on Saturday February 19, 2022 @12:29AM (#30394666)
More like 24 no? 12 until it is deprecated then 12 of support.
Re:Kubernetes: The Documentary by dastbe on Sunday February 13, 2022 @02:52PM (#30324971)
if you have the right company setup, headcount, and culture then you can absolutely build a world where teams can be allowed to force “perfection” over time, even with amazing levels of automation.

i think the interesting thing is that while you’d think that is the ideal everyone should strive for, the reality is that’s inapplicable or detrimental to run most businesses that way.

Re:Kubernetes: The Documentary by oblio on Sunday February 13, 2022 @03:36PM (#30325397)
> build a world where teams can be allowed to force “perfection” over time, even with amazing levels of automation.

I'm confused, did you mean "without" instead of "with"?

Also:

> i think the interesting thing is that while you’d think that is the ideal everyone should strive for, the reality is that’s inapplicable or detrimental to run most businesses that way.

Why would you say that is?

Re:Kubernetes: The Documentary by emodendroket on Sunday February 13, 2022 @04:09PM (#30325713)
I think we see a good reason here: normal people don't think "this API is so beautiful," they think "these clowns are making me do a migration yet again."
Re:Kubernetes: The Documentary by dastbe on Sunday February 13, 2022 @04:08PM (#30325704)
i meant with. google has built some really great tools for minimizing toil when pushing major changes across a codebase. even then, there’s a nontrivial overhead that is borne across the company.

> Why would you say that is?

many/most businesses don’t have the margins/scale/time to focus on it. in a lot of businesses (amazons included) the goal for the business is to try as many things as fast as possible and scale that which works and abandon what fails. the goal of infrastructure there is to get out of the way and enable the layers above to build things as naively and simply as possible with little to no churn.

Re:Kubernetes: The Documentary by servytor on Sunday February 13, 2022 @04:32PM (#30325887)
It is because Amazon dog-fooded their own technology stack. This forced them to have a better interface to their infrastructure, which in turn made it easy to open to the public. Google on the other hand, hid behind their own complexity and let engineers feel stupid for not knowing Borg like the back of their hand.
Re:Kubernetes: The Documentary by sorry_outta_gas on Sunday February 13, 2022 @02:41PM (#30324882)
Google is poor at support, web hosting/cloud is as much about support as it is technology
Re:Kubernetes: The Documentary by hackandtrip on Sunday February 13, 2022 @03:09PM (#30325125)
In my little experience, although GCP docs leaves a lot to be desired, the support was always very thorough and fast to answer, even for not critical/emergency situations.
Re:Kubernetes: The Documentary by pm90 on Sunday February 13, 2022 @09:28PM (#30327785)
One of the frustrating thing with their docs is that they keep changing them without versioning (last time I checked). So it’s easy to feel like you’re being gaslighted because assumptions you made on the basis of the docs could have changed and then you look stupid.

AWS being the dominant player means that if they make any silent changes there will be a SO/GitHub issue/HN thread about it. But with GCP without their pricy professional tier support you can feel a bit unsure and lost all the time about your understanding of it.

Re:Kubernetes: The Documentary by sorry_outta_gas on Sunday February 13, 2022 @03:48PM (#30325531)
They have improved a lot recently
Re:Kubernetes: The Documentary by paulluuk on Sunday February 13, 2022 @03:15PM (#30325168)
To be honest, AWS cloud support isn't that fantastic either. Responses to tickets usually come in 48 hours, if not more.

I don't know about you, but in 48 hours I've usually already built a workaround and moved on to other projects.

Not saying that AWS support is worse or even equal to GCP support, but just commenting that cloud support in general feels pretty useless and an afterthought at best.

And let's not even get started on the horrible, awful UI in AWS console.

Re:Kubernetes: The Documentary by jethro_tell on Sunday February 13, 2022 @06:32PM (#30326705)
You have to pay for support. I've found it to be one of the best. I've open a ticket in the portal with a detailed issue before I go to lunch and have a level 3 tech call me before I get back.

I've had this happen multiple times for multiple services and the person's i was able to work with had enough knowledge and access to either see detailed logs and walk me through some undocumented feature or file a bug report by days end.

I'm spoiled in that regard but I've never had non-enterpise support with anything other than the 4 hour sla.

Re:Kubernetes: The Documentary by p_l on Sunday February 13, 2022 @07:54PM (#30327272)
It can really depend. I know a lot of stories of AWS support being stellar while google was bad (even discounting support for non-GCP stuff from the list).

But at the same time I can refer to enterprise-tier support at AWS, which has gone past ticket stage into "regular calls with AWS support engineers to help architect our solution" and certain critical topics ended up ghosted (my first reaction was that support had no idea about the functionality we asked about, later experienced taught me that at best, the AWS documentation lied by omission. At worst it was divorced from reality.)

Re:Kubernetes: The Documentary by grogenaut on Sunday February 13, 2022 @04:44PM (#30325983)
pay for enterprise support, response is within minutes
Re:Kubernetes: The Documentary by debuggerpk on Sunday February 13, 2022 @11:27PM (#30328363)
I think Google did .. App Engine came around 2008.