News for hackers. Stuff that matters.
Re:Train Your AI Model Once and Deploy on Any Cloud by politelemon on Saturday July 08, 2023 @06:23AM (#36643061)
I'm failing to see why k8s needs to be involved here - it's overkill for most model serving cases but its involvement here now adds additional overhead. So it's not really any cloud, it's any cloud where you're running your EKS/AKS etc.
Re:Train Your AI Model Once and Deploy on Any Cloud by ianpurton on Saturday July 08, 2023 @06:42AM (#36643160)
Kubernetes means you don't have to learn each clouds way of doing a deployment. You just learn the k8s way then use that with Google, Azure or whatever.

So your skillset is reusable.

Re:Train Your AI Model Once and Deploy on Any Cloud by finikytou on Saturday July 08, 2023 @06:46AM (#36643178)
no it doesnt mean that. you still need to know how to operate k8s on a specific cloud.
Re:Train Your AI Model Once and Deploy on Any Cloud by jml78 on Saturday July 08, 2023 @07:40AM (#36643465)
Just like you have to know how to operate each cloud in general.

There is no free lunch. But if you learn k8s, moving from AWS EKS to Google GKE to DigitalOceans hosted k8s is easy.

Re:Train Your AI Model Once and Deploy on Any Cloud by quickthrower2 on Saturday July 08, 2023 @06:54AM (#36643225)
hell yes.
Re:Train Your AI Model Once and Deploy on Any Cloud by hhh on Saturday July 08, 2023 @08:44AM (#36643858)
Ideally the developer doesn’t. At scale some platform or infra team should.
Re:Train Your AI Model Once and Deploy on Any Cloud by artdigital on Saturday July 08, 2023 @08:53AM (#36643908)
I run stuff on hosted k8s from DO, Google and Vultr. I can absolutely reuse my knowledge, and deployments are almost identical (minus smaller differences like storage csi driver, etc)
Re:Train Your AI Model Once and Deploy on Any Cloud by oceanplexian on Saturday July 08, 2023 @11:20AM (#36645251)
I work at a place running a million containers deployed in all 3 (Azure, AWS, GCP). I can assure you they are radically different; autoscaling works differently, the load balancers work differently, the networking infrastructure is completely different, the failure modes and limits behaviors are different, the instances perform differently, observability is different, and they all suck in unique and different ways that we discover on a daily basis. Shit even AWS can't keep their regions consistent; each region has different products and features and they fail in different ways.

If you are the one maintaining it it's a full time job handling all these edge cases, it's completely miserable and I wouldn't recommend it to anyone.

Re:Train Your AI Model Once and Deploy on Any Cloud by Infernal on Saturday July 08, 2023 @11:52AM (#36645533)
> I work at a place running a million containers deployed in all 3 (Azure, AWS, GCP)

Are you using AKS, EKS, GKE on those providers, or deploying your own k8s on top of the compute those providers offer? It sounds to me like the former.

Re:Train Your AI Model Once and Deploy on Any Cloud by hosh on Saturday July 08, 2023 @12:50PM (#36646189)
I’ve done smaller deployments on GKE and another on EKS, and I can tell you, they are different enough. It’s when you start having to autoscale, optimize resources by instance types, and manage network ingress that these quirks start really come out. The essential ideas are invariant across cloud providers though.

But I enjoy working with Kubernetes.

Re:Train Your AI Model Once and Deploy on Any Cloud by Infernal on Saturday July 08, 2023 @06:55PM (#36649600)
I should’ve been clearer about what I was getting at. I agree AKS, EKS, GKE etc (cloud gnostic k8s) are different enough to cause a growth of complexity when managing a mixed environment of them.

The post I was replying to seemed to be saying (by analogy) “Linux is hard to manage because I run into all sorts of trouble trying to support a mixed environment of SuSE, Ubuntu and RHEL, therefore Linux is just too complicated”.

Re:Train Your AI Model Once and Deploy on Any Cloud by gymbeaux on Saturday July 08, 2023 @02:40PM (#36647422)
The latter wouldn’t make sense. In such a case, there’s already little value to being in the cloud, but to be in several?..
Re:Train Your AI Model Once and Deploy on Any Cloud by floomk on Saturday July 08, 2023 @04:50PM (#36648631)
Once you get to the scale of a million containers (or "apps") spread over multiple clouds then everything will be miserable.
Re:Train Your AI Model Once and Deploy on Any Cloud by hosh on Saturday July 08, 2023 @12:45PM (#36646134)
The essential ideas that Kubernetes exposes concretely are invariant across cloud providers. There absolutely are nuances and quirks that are different for each cloud provider, and unique for the workload you have. However, those same ideas also act as a kind of mental framework in which these quirks can be understood from. It isn’t as if those quirks are randomly there, unconnected to anything, and therefore not part of a coherent design.

For example, the consistent use of labels as a way to identifying groups of resources that need to coordinate with each other is very useful for any distributed system. I find myself looking for them in say, CI/CD systems (in the form of agent tags), or at the application level in say, matching players to game servers.

Re:Train Your AI Model Once and Deploy on Any Cloud by Art9681 on Saturday July 08, 2023 @08:56AM (#36643938)
No you don't have to. You can deploy your own cluster instead of using the managed option if you want to. A good SRE can deploy and manage EKS. A great SRE can deploy and manage a cluster to any Cloud without ever touching the dashboards.
Re:Train Your AI Model Once and Deploy on Any Cloud by okamiueru on Saturday July 08, 2023 @10:55AM (#36644991)
Some part of it creates cloud specific resources, and you might also for good reasons have cloud manged database or data storage that your k8s services use. However, "k8s on a specific cloud", is mostly the same, except for the outer edges.
Re:Train Your AI Model Once and Deploy on Any Cloud by hosh on Saturday July 08, 2023 @12:58PM (#36646270)
Until you want to scale and optimize resources.

I enjoy working with Kubernetes, but forcing a complex domain into something legible is a recipe for catastrophe. There are quirks, across cloud providers, and this is just another day in Ops, with or without Kubernetes. (See: https://www.ribbonfarm.com/2010/07/26/a-big-little-idea-call... )

Re:Train Your AI Model Once and Deploy on Any Cloud by api on Saturday July 08, 2023 @10:57AM (#36645002)
It’s more like learning different Linux distributions than learning different OSes.
Re:Train Your AI Model Once and Deploy on Any Cloud by jcims on Saturday July 08, 2023 @10:10AM (#36644576)
Kubernetes gets a lot of shit on HN but for all of its challenges it has proven to be a fantastic method to abstract many of the idiosyncrasies of hosting on-prem vs various cloud providers. I've worked for two companies now with 8 figure monthly cloud spend and hundreds/thousands of applications operating in one or more of the main cloud providers and k8s has been essential in making that happen. Teams can migrate to an on-premise hosted option if they want, then transition to cloud if/when it makes sense, or just stay where they are.
Re:Train Your AI Model Once and Deploy on Any Cloud by stavros on Saturday July 08, 2023 @12:15PM (#36645797)
Takes notes

"Kubernetes has been essential in making 8-figure monthly cloud spend happen"

Re:Train Your AI Model Once and Deploy on Any Cloud by windexh8er on Saturday July 08, 2023 @12:28PM (#36645932)
The last startup I was in was obsessed with putting everything into k8s for no apparent reason. Even the product they were selling, which most customers hated because it forced our customer base to either have to deal with the pain of paying for k8s because they had no need or intention to use it other than for our product or work with a cross-functional team which now created a time sink and dependency that wouldn't have been there otherwise.

The best though was when I ran across someone in the org trying to run a single container to run a periodic job in its own cluster. They spent half the day trying to get it to work with ingress.

You can imagine how it came to a head when the company realized they were spending hundreds of thousands per month on idle clusters in AWS.

Re:Train Your AI Model Once and Deploy on Any Cloud by immibis on Saturday July 08, 2023 @01:47PM (#36646793)
One company I worked at was obsessed with k8s for a while, on a local arrangement of about 4 servers, each build would start up a new container on kubernetes and rebuild an entire operating system from scratch.
Re:Train Your AI Model Once and Deploy on Any Cloud by immibis on Tuesday July 11, 2023 @10:26AM (#36681438)
clarification: the thing being built was an operating system. It didn't build an operating system to run the real build on.
Re:Train Your AI Model Once and Deploy on Any Cloud by jcims on Saturday July 08, 2023 @01:17PM (#36646431)
When you’re spending billions of dollars on data center refresh, it’s a bargain.
Re:Train Your AI Model Once and Deploy on Any Cloud by oceanplexian on Saturday July 08, 2023 @11:12AM (#36645156)
> Kubernetes means you don't have to learn each clouds way of doing a deployment.

So instead of learning how to deploy on GCP, AWS, and Azure, which is only 3x more complicated than deploying to a single cloud, you should learn K8s, which is 10-15x more complicated, in addition to still having to learn about all the various ingress controllers and weird quirks that are completely different on each cloud provider. Doesn't really track for me.

Re:Train Your AI Model Once and Deploy on Any Cloud by echelon on Saturday July 08, 2023 @11:38AM (#36645406)
> which is 10-15x more complicated

You can learn k8s in a day. It's really simple.

> various ingress controllers and weird quirks that are completely different on each cloud provider

Which are thoroughly documented and not that hard to implement or understand. You'd be reading about each cloud's nonstandard ingress even without k8s.

The beauty of k8s is you can run your software locally and have a much easier time lifting and shifting to another cloud.

Fitting to the shape of a cloud provider is a great way to never leave.

Another benefit of k8s is that you treat your services as cattle you can easily spawn and kill. Adoption of k8s naturally leads to anti-fragility, anti-brittle best practices.

Re:Train Your AI Model Once and Deploy on Any Cloud by hosh on Saturday July 08, 2023 @12:36PM (#36646049)
I think being able to abstract the cloud providers is secondary to k8s’s ability to self-heal and its modularity.

But I think most developers don’t care, and instead, should interact with a platform built with Kubernetes as a foundation.

Re:Train Your AI Model Once and Deploy on Any Cloud by profunctor on Saturday July 08, 2023 @04:33PM (#36648442)
Maybe this is because I’m not that smart but I could not learn real kubernetes in a day. I had to build a system for loading models and returning predictions over a HTTPS api. It had to connect to storage to load the model, needed secrets etc. It took more than a day. And I think it would take most people more than a day to go from zero to able to create a useful, real world deployment in a day. I’m sure you can rush through the documentation in a day but I wouldn’t call that learning.
Re:Train Your AI Model Once and Deploy on Any Cloud by bg24 on Saturday July 08, 2023 @06:54PM (#36649592)
Having spent 4 years working with kubernetes (though as a PM, but pretty hands-on), getting started is easy - like in less a week. The problem happens when you run into issues. That can suck up lot of time. Also if you are new to containers, it might not be a good step to venture into k8s.
Re:Train Your AI Model Once and Deploy on Any Cloud by hosh on Saturday July 08, 2023 @12:33PM (#36646013)
K8S is not that complicated. Once you know the big ideas behind it, and how to reason with it, it becomes a very versatile platform substrate.

Probably the biggest one is understanding you don’t ever do anything directly with Kubernetes.

Re:Train Your AI Model Once and Deploy on Any Cloud by el_benhameen on Saturday July 08, 2023 @01:37PM (#36646677)
Do you have any favorite learning resources?
Re:Train Your AI Model Once and Deploy on Any Cloud by doctoboggan on Saturday July 08, 2023 @03:58PM (#36648135)
I've learned K8S over the past few months and what was absolutely instrumental to my understanding was: 1. use Helm, and 2. daily chat sessions with gpt-4.

I use gpt-4 through the API where you can set your own system prompt. I developed one that basically instructed it to give me kubectl commands to solve my problems and then wait for me to give it the result before continuing. Through this I learned the practical techniques and which kubectl commands you use on a daily basis, which is so much more helpful than reading the documentation which just gives all commands equal weight.

EDIT: Oh, and definitely watch a few "TechWorld with Nana" videos on YouTube. She does a great job of explaining the architecture, terminology, and philosophy of k8s which I think it very helpful to know.

Re:Train Your AI Model Once and Deploy on Any Cloud by hosh on Sunday July 09, 2023 @09:38AM (#36654360)
I learned k8s the hard way so I am not sure I can point those out.

That mindset of not doing anything directly comes from understanding that you are not setting up pods up, but rather, you are setting automated processes up. These processes knows what the desired state is, and if there is something that changes things, it takes actions to get back to that desired state.

So you don’t create pods. You create deployments that maintain a set of pods. You don’t assign pods to nodes. You define pod affinities, and use node selectors on nodegroups. You use pod priority. You make graceful startups and shutdowns work correctly. And so forth.

Re:Train Your AI Model Once and Deploy on Any Cloud by saurik on Sunday July 09, 2023 @04:02AM (#36652570)
Then what are you using, if not kubernetes, when you are using kubernetes?
Re:Train Your AI Model Once and Deploy on Any Cloud by hosh on Sunday July 09, 2023 @09:44AM (#36654403)
You don’t use kubernetes. You’re defining desired states for various processes that are collectively called “Kubernetes”. These automated processes maintain things at the desired state, and take action to move things back to the desired state when that happens. These processes are distinct, and are not usually aware of each other (example: horizontal pod autoscaling and cluster autoscaler).

If you understand that, you then know you can add processes (such as operators or the cluster autoscaler).

It’s a mindset shift. Thinking of it as “using” kubernetes, as if it is a monolithic thing that you directly control, will greatly increase the difficulty in understanding and reasoning through what’s going on within a Kubernetes cluster.

Re:Train Your AI Model Once and Deploy on Any Cloud by ithkuil on Sunday July 09, 2023 @04:15AM (#36652638)
A lot of people conflate the pains of installing and managing the kubernetes "infrastructure", with the pains of deploying and managing an application running on kubernetes.

The complaints are real, because in practice a company needs both aspects and when a small company struggles to setup and manage the kubernetes infrastructure correctly, the application operators are suffering the consequences (e.g. log collection infrastructure doesn't work, it's hard to provision nodes with the right capacity, things like that). They see the infrastructure operation team struggle and they partake in that struggle because what is advertised to be easy, it's not.

That said, K8s is a very good way to build an "API" between infrastructure and application "teams". It can work very well if the people involved set things and processes up correctly. It can be a nightmare if botched up

Re:Train Your AI Model Once and Deploy on Any Cloud by hosh on Sunday July 09, 2023 @09:29AM (#36654302)
> That said, K8s is a very good way to build an "API" between infrastructure and application "teams". It can work very well if the people involved set things and processes up correctly. It can be a nightmare if botched up

Agreed.

Someone has to actually build it with a product mindset — including product-market fit. I’m one of those oddballs that have set up and scaled up infra, and have put together and delivered applications before.

In one gig, I had people joke about naming what I had setup after Heroku. I didn’t realize its significance until I came to a place where it was not done this way. Many on the application team have expressed dissatisfaction… but it’s like the infra team is oblivious to that.

Re:Train Your AI Model Once and Deploy on Any Cloud by beebmam on Saturday July 08, 2023 @05:38PM (#36649028)
No shot that kubernetes is 10-15x more complicated than cloud offerings.
Re:Train Your AI Model Once and Deploy on Any Cloud by ChatGTP on Sunday July 09, 2023 @07:50AM (#36653711)
There is no way k8s is that complicated? Not sure you’ve used it in anger but it is awesome.
Re:Train Your AI Model Once and Deploy on Any Cloud by berkle4455 on Saturday July 08, 2023 @12:05PM (#36645674)
This argument is reminiscent of ORMs and “you can switch your database and only change the config!”

Switching your database, just like switching your cloud provider, rarely happens in practice.

Re:Train Your AI Model Once and Deploy on Any Cloud by danryan on Saturday July 08, 2023 @12:18PM (#36645836)
That feature was always a byline at best.
Re:Train Your AI Model Once and Deploy on Any Cloud by bjornsing on Saturday July 08, 2023 @07:10AM (#36643317)
> So it's not really any cloud, it's any cloud where you're running your EKS/AKS etc.

As I understand it this new Nvidia VM image comes with Kubernetes on the inside so to speak, perhaps microk8s with nvidia extension enabled.

BTW this is how I’ve started running my own little AI experiments. Sure, there’s some overhead. But compared with constantly downloading new versions of drivers it’s quite lightweight. Also K8S is turning into the ligua franca of sodtware platforms, so well worth learning and paying the overhead on IMHO.