News for hackers. Stuff that matters.
Re:Rolling your own servers with Kubernetes by argd678 on Monday May 06, 2019 @11:28AM (#19840293)
Last physical datacenter my team at the time ran our app in we spent weeks troubleshooting some hardware network driver issues that caused the network to drop. Was an enormous distraction and Dell and VMware support were useless at resolving it for us. I’m glad to have others deal with those low level “oh it must just be your setup” issues.
Re:Rolling your own servers with Kubernetes by lima on Monday May 06, 2019 @11:37AM (#19840358)
Anecdotal evidence. I've also seen colleagues spend weeks arguing with AWS support while debugging a weird performance degradation issue, that would have been straight-forward to investigate in a bare metal deployment with full control over everything.

It's not like the cloud is a magical place where no unexpected issues ever happen. Cloud providers can be surprisingly buggy, especially AWS, particularly at scale when you start hitting the implicit scaling limits their docs conveniently forgot to mention.

Each technology has their unique challenges.

Re:Rolling your own servers with Kubernetes by falcolas on Monday May 06, 2019 @12:13PM (#19840749)
> spend weeks arguing with AWS support

Yup. We've run into that repeatedly. The "we didn't notice anything on our side, please send more screenshots and logs" gets really old when working with "managed services".

Network packet losses/truncations, EKS control plane failures, cloudformation stacks getting stuck in really wierd states, inconsistent cloudformation implementations for new and existing services, ENI weirdness in containers...

Managed services just feel like they aren't - more and more every day. Amazon (or any other cloud) will never have the same investment in your availability and infrastructure as you will.

Re:Rolling your own servers with Kubernetes by ReidZB on Monday May 06, 2019 @03:48PM (#19843054)
My experience with AWS support for performance issues is similar. My ticket for an Aurora RDS performance issue ended up in a kind of purgatory where they continued to ask for more information and more tests on my side, continued, to infinity. Once one support engineer was satisfied with my results, the ticket would go dark for months, and when I would follow up, another (new) engineer would ask for a whole new round of tests, logs, etc., typically for the latest release.

After a few rounds of back-and-forth, I eventually gave up.

(I even gave them permission to copy test snapshot data, at their request, and sample queries to reproduce the issue. They could have done all the testing they wanted in-house. But that might have actually required them to spend time on their side to diagnose and fix something, I guess.)

It wasn't a critical issue, exactly, and we worked around it in code. I mostly just wanted to report a problem.

Re:Rolling your own servers with Kubernetes by rabi_penguin on Monday May 06, 2019 @12:39PM (#19841078)
Running into one of those exact issues (cloudformation stacks getting stuck in really weird states) right now. A real pain in the ass, because I have zero control or visibility into it.
Re:Rolling your own servers with Kubernetes by toomuchtodo on Monday May 06, 2019 @01:02PM (#19841374)
Is switching to Terraform an option?
Re:Rolling your own servers with Kubernetes by falcolas on Monday May 06, 2019 @02:21PM (#19842234)
Based on our own research - probably. But it requires tooling to integrate into a CI/CD pipeline, conversion of existing CF stack definitions to Terraform definitions, and a lossy/downtime laden migration.

For a team with the goals of IAC, hands-off, and high-availability infrastructure, migrating is a pain.

Re:Rolling your own servers with Kubernetes by ReidZB on Monday May 06, 2019 @03:53PM (#19843109)
With Terraform's import functionality, you might be able to migrate into Terraform without any downtime. I believe there are even utilities to translate CloudFormation code straight into Terraform, though I've never tried them.

It would definitely be an annoying migration, even if import works for your use-case, but honestly, dealing with CloudFormation is so irritating on a day-to-day basis for me that I'd consider it worth it.

Re:Rolling your own servers with Kubernetes by isatty on Monday May 06, 2019 @12:48PM (#19841185)
Cloudformation is a tragedy in itself.
Re:Rolling your own servers with Kubernetes by falcolas on Monday May 06, 2019 @02:24PM (#19842271)
The idea behind CloudFormation is great, though. Platform-native IAC with promised first party support for all future projects, and backfilling existing services. Plus, it supports deep integrations with first-party supported configuration management services.

The problem is that the reality has not lived up to the promise, and "first party support" means "only the first party can support".

Re:Rolling your own servers with Kubernetes by devonkim on Tuesday May 07, 2019 @10:30AM (#19849714)
CloudFormation is a declarative state management language and framework just like Terraform. The problem is that CF abstracts said state away from you to the degree that you can't hack around it and you wind up forcibly deleting resources if you try to use it like a configuration management framework. With CF Custom Resources you can add all sorts of other stuff and that's pretty cool at least.
Re:Rolling your own servers with Kubernetes by TheCraiggers on Monday May 06, 2019 @01:27PM (#19841687)
It's infinitely better than what Azure provides, IMO.
Re:Rolling your own servers with Kubernetes by tech_tuna on Monday May 06, 2019 @10:38PM (#19845927)
I am a Terraform convert, I agree.
Re:Rolling your own servers with Kubernetes by john_moscow on Monday May 06, 2019 @01:46PM (#19841889)
You need to treat a commodity like a commodity. It's perfectly OK to run business-critical processes in someone's cloud, as long as you have a fully automated process that will deploy your entire environment from the latest backup to any set of cloud nodes at any cloud provider you point it to. Unless you're doing that, you're just shifting risks, not eliminating them.
Re:Rolling your own servers with Kubernetes by wazoox on Monday May 06, 2019 @12:06PM (#19840673)
Yup, the famous "leaky abstractions"[0]. You can make some reliable assumptions of the data rate of a 40Gbps link in your DC between this and that physical server, and how to get the best out of it. On AWS? Not so much.

0: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...

Re:Rolling your own servers with Kubernetes by alexnewman on Monday May 06, 2019 @11:51AM (#19840506)
I've absurd bugs at AWS,GCP and azure. I love dynamic capacity but clouds are super flakey, especially AWS. Azure networks are the worst. GCP I haven't used in over a year so I wont talk ...
Re:Rolling your own servers with Kubernetes by argd678 on Monday May 06, 2019 @11:40AM (#19840395)
I’ve had performance issues with GCP, however I can switch cloud providers in that case without too much effort. With physical hardware that’s much harder.
Re:Rolling your own servers with Kubernetes by lima on Monday May 06, 2019 @11:59AM (#19840605)
If you have a well-built and automated bare-metal deployment, it's no harder than switching cloud providers. At the scale where it makes sense to go bare metal, either would be a lot of effort.

An extra abstraction layer like k8s makes it a lot easier, which is exciting. It's also precisely why IBM bought Red Hat - a well-built k8s distribution like OpenShift is one of the very few real alternative to public clouds for many companies.

Re:Rolling your own servers with Kubernetes by windexh8er on Monday May 06, 2019 @12:29PM (#19840965)
> An extra abstraction layer like k8s makes it a lot easier, which is exciting. It's also precisely why IBM bought Red Hat - a well-built k8s distribution like OpenShift is one of the very few real alternative to public clouds for many companies.

OpenShift is built on a prescriptive mentality. They choose everything for you from OS through pipeline and many deviations are unsupported. As you mentioned there are others, but depending on your environment OpenShift is either a very good fit or square-peg-round-hole.

Re:Rolling your own servers with Kubernetes by lima on Monday May 06, 2019 @01:12PM (#19841508)
OpenShift is a superset of k8s, so you're free to customize it (you just won't get support if it breaks).
Re:Rolling your own servers with Kubernetes by humbleMouse on Monday May 06, 2019 @01:56PM (#19841984)
Openshift doesn't choose the pipeline. You can deploy to it however you want. Openshift is literally the same exact thing as kubernetes, with a simpler api and a nice ui. Openshift is extremely flexible and you can use it however you want. Don't know why you think it's so restrictive.
Re:Rolling your own servers with Kubernetes by lima on Monday May 06, 2019 @02:55PM (#19842568)
It has the same API as k8s. It's literally the same thing, OpenShift just adds some PaaS feature on top of k8s.
Re:Rolling your own servers with Kubernetes by snuxoll on Monday May 06, 2019 @04:05PM (#19843226)
Which is precisely why I've run it for over 2 years now. The devs on my team love throwing a git URL into OpenShift and have a project go live in mere minutes, previously a more senior developer would need to setup a CI pipeline and deployment automation and even with experience it was a chore.

It's not like I'm personally afraid of k8s or building docker images myself either, but anything to help my team be more productive without needing to hold hands or do gruntwork on their behalf improves the lives of everyone.

Re:Rolling your own servers with Kubernetes by lima on Monday May 06, 2019 @04:28PM (#19843488)
Exactly right. I don't understand all those teams reinventing the wheel building custom k8s distributions, rather than using something like OpenShift and focusing on their core business.

Running custom k8s in production is the 2019 equivalent of using Gentoo.

Re:Rolling your own servers with Kubernetes by argd678 on Monday May 06, 2019 @04:06PM (#19843249)
You’re forgetting the network needs to be replaced too, there’s lead times on getting hardware, there the whole racking and cabling, and even at that if the this is due to a more complex networking issue then I can’t swap out the network since there are too many components and they’re shared by other teams. In the cloud I can do this quickly tho.
Re:Rolling your own servers with Kubernetes by dilyevsky on Monday May 06, 2019 @11:53AM (#19840527)
You wait until you have to debug something like that with a VPC. We had to spend couple days to convince our cloud support the issue was on their end during full site outage.
Re:Rolling your own servers with Kubernetes by ori_b on Monday May 06, 2019 @12:05PM (#19840667)
Counterpoint: These issues still exist, you just lose visibility into them by putting someone else in between. And usually, the someone else is big enough that you're not a priority.
Re:Rolling your own servers with Kubernetes by nemonemo on Monday May 06, 2019 @12:13PM (#19840755)
with limited resource, you have to choose to balance between the stability or more control over your infra. IMO, small companies can't afford both, but also likely they would not need both at that size.
Re:Rolling your own servers with Kubernetes by rhizome on Monday May 06, 2019 @03:22PM (#19842802)
B.S. I was solo IT for a 50+ppl company, on a three-person team with a CIO/DBA and a data manager. I supported a full Windows/Exchange LAN with heavy printing needs, tape backups, internal/external infrastructural services and webservers with LOB java webapps, remote users, etc. Through all this I learned Perl from scratch, rearchitected the business systems (hardware) and reduced their production time by like 90% (two weeks to two days) with zero downtime (primordial blue-green, as it turns out), and wrote a completely new ETL flow with my new Perl skills. From four servers under a table to four racks, SaaS architecture is exceedingly simple next to all this, and the company spent probably around $10K/mo all told, maybe up to $20k while I was buying hardware.

Maybe you simply haven't seen this kind of operation in action before?

Re:Rolling your own servers with Kubernetes by closeparen on Monday May 06, 2019 @11:30AM (#19840315)
At the scale where owning infrastructure makes sense, that's not a distraction but some team's full-time mission.
Re:Rolling your own servers with Kubernetes by falcolas on Monday May 06, 2019 @12:18PM (#19840823)
Even before that tipping point, you're likely to have a team dedicated to working with infrastructure hosted by a cloud provider. Based off past (hardware) and current (AWS) experience, the team sizes aren't that different.
Re:Rolling your own servers with Kubernetes by argd678 on Monday May 06, 2019 @11:55AM (#19840550)
We did actually have a team who handles hardware and a networking department, and they couldn’t figure it out. So lots of people had to get involved to do a binary search for the problem. I should mention this is a company with considerable resources.
Re:Rolling your own servers with Kubernetes by nemonemo on Monday May 06, 2019 @11:38AM (#19840371)
In my last team, our team suffered from a CPU bug and a SSD bug that caused painful debugging nights. We used multithreading heavily, and it could have been library/kernel issues or some bugs in our code, which cover massive amount of potential target code that needed to sort through initially. I don't think small companies can afford such gigantic debugging effort, so either they should rely on their luck (or flaky tests) or on someone else who can afford it (a cloud/infra service provider or a startup that depends on you.)
Re:Rolling your own servers with Kubernetes by lima on Monday May 06, 2019 @11:41AM (#19840401)
How is that related to bare metal vs. clouds? That could have happened on either.
Re:Rolling your own servers with Kubernetes by nemonemo on Monday May 06, 2019 @11:52AM (#19840518)
Big cloud providers or some big VC backed companies have the resource to figure out the issues by wrestling with the gigantic and unyielding vendors, or by working around it with their engineering resource. Not many teams have the time for even wanting to understand the issues. This is not an inherent problem of bare metal, but one aspect to consider if you were to choose the path. But you are right, cloud providers have their own quirks.
Re:Rolling your own servers with Kubernetes by nocman on Monday May 06, 2019 @12:55PM (#19841275)
> Big cloud providers or some big VC backed companies have the resource to figure out the issues by wrestling with the gigantic and unyielding vendors, or by working around it with their engineering resource.

This is all operating on the assumption that those entities will care enough about your problem enough to do something meaningful to fix it. As others have stated here, at least in the context of "Big cloud providers", that frequently is not the case. Often you just get the runaround (continuous delays, requests for "more information", other attempts to stall in first level support).

Add a third party "managing" some piece of your company's infrastructure via a cloud provider into the mix and it often gets even worse.

It is true that there are real costs to having your own hardware/software onsite and people who know how to manage it. However, the promised reductions in cost/hassle of moving things offsite are frequently offset or even exceeded by the costs/hassles you get by not having control of things yourself.

Re:Rolling your own servers with Kubernetes by syn0byte on Monday May 06, 2019 @02:09PM (#19842110)
Except they won't. Not for an esoteric or exotic bug that doesn't affect 99.9% of their customers. Unless your businesses contract costs more than that team of technical and engineering resources(Eg your Netflix et al), it's not worth their time to bother looking into it.

All that assumes they even admit the issue is their hardware not your code, which is a mighty big assumption itself.

Re:Rolling your own servers with Kubernetes by Nullabillity on Monday May 06, 2019 @01:01PM (#19841354)
Cloud providers also tend to run much more exotic configs for pretty much everything (since tenant isolation is a top priority), so such issues are more likely to happen in the first place.
Re:Rolling your own servers with Kubernetes by turk73 on Monday May 06, 2019 @01:07PM (#19841430)
Because when you're caught between those who manage the hardware and those who manage the K8s cluster, you get ping-ponged between them in the blame shifting. It's very annoying.
Re:Rolling your own servers with Kubernetes by qaq on Monday May 06, 2019 @11:34AM (#19840340)
Until you hit this can not happen we will not spend time looking into it issue in a cloud :)