News for hackers. Stuff that matters.
Re:Back to the '70s with Serverless by mlthoughts2018 on Saturday December 19, 2020 @10:15PM (#25483522)
I find your quote here and the follow up seriously unconvincing. You know what a miserable workflow is? Preparing all my build scripts according to the half baked build tool that our in-house Deployment team cooked up to use fabric to spray my deployment onto a bunch of nodes in our data center, all the tests and script linting pass, so then I go push it through our CI/CD pipeline (which can’t do the above tests for me because again this is all threadbare maintained by in-house tooling teams who could not be less aligned to end product deliverables), and lo and behold it breaks. Good thing I’m not paying so much for CloudWatch so I get to literally go ssh to the worker node of the CI/CD system where my task failed and cross my fingers that it wasn’t reclaimed for the next task and that the log files weren’t rotated off, grep my way to tracebacks that probably involve things no one on my team has ever heard of, spend 1/2 a day chasing them all down (if I’m lucky), and finally kicking the box to restart a build a few times and it goes. Then it goes to the in house homebrewed kubernetes deployment scripts where it promptly chokes on a secret that wasn’t configured right (thanks home made script linters), so then I have to go down a rabbit hole of eye melting Helm config of secrets for an hour but thank god I’m not paying for a simple cloud UI for this. So we get the secret cleared up and everything is great, until the kubernetes scheduler can’t schedule my pods because our data center is at capacity and now I literally can’t get work done until I chase down some probably incorrect and certainly unmaintained Grafana chart of who is using how much excess memory on which nodes, and wrangle someone with authority to force another team to stop eating lunch and downsize pods right now. After all this, 6-7 hours later and 20-30 labor hours later, my workload is finally deployed, but it will die overnight for an uncaught reason that doesn’t alert anyone, and even if it did, the team managing compute resources is in a permanent state of PTSD from never ending pages so they wouldn’t bother caring about why my workload failed and will just assume it’s application logic bugs or Docker bugs on my side for as long as they can just to avoid yet another un-debuggable failure landing on their plate, so that’ll be great for morale in the morning when I get to derail yet another day and spend time doing every type of ops work that is not my job and prevents us from delivering product goals.

This is every company I’ve ever worked for or ever heard any colleagues, friends or associates work for, that tries to build their own SRE practice on top of data centers.

Large cloud platforms alleviate so so much of this pain. The “back to the 70s” workflow sarcastically described in the quoted passage you shared sounds like an absolute dream compared to the nightmare of shipping code on internally created developer tool & deployment platforms interfacing to bare metal or space leased from data centers.

I can’t give the cloud providers my money fast enough and I have only ever felt happy about that for years and years now.

Re:Back to the '70s with Serverless by TheColorYellow on Sunday December 20, 2020 @02:17AM (#25484564)
This is the type of cultural accuracy I've been looking for.

Are all the commenters in this thread just cynical engineers who really can't see past their own narrow world view? I get snake oil is snake oil, but there is such a strong brand of "anything new must be snake oil" that runs through the HN community it boggles my mind.

The world is actually full of complex systems. And only once these systems are in place does the software engineer get to do his work! People on this site act as if they have all the understanding and knowledgeable necessary to decry something like cloud vendors because clearly these same people know better than everyone else and these are actually simple problems if only we had their perspective and knowledge.

The ego and arrogance of people in tech really shines through here.

Re:Back to the '70s with Serverless by nocman on Sunday December 20, 2020 @03:20AM (#25484779)
> there is such a strong brand of "anything new must be snake oil" that runs through the HN community.

I would say there's at least as strong a "brand" of "if it is a new way it is probably a better way" on HN as well.

I think making blanket assumptions in either direction is a terrible idea. Best to be aware of your own biases, and try to be as objective as you can in your decisions.

Experience matters, and there are new solutions that are inferior to older ones in many cases (yes, there is plenty of "snake oil" for sale). However, sometimes new people look at an old problem with fresh eyes and come up with better ideas on how to solve it.

This really isn't anything new. When I was fresh out of college I remember older programmers talking about "such-and-such" is just the old way of doing it repackaged. There is lots of oscillation in the computer industry.

Re:Back to the '70s with Serverless by papaf on Sunday December 20, 2020 @03:04AM (#25484726)
People on this site act as if they have all the understanding and knowledgeable necessary to decry something like cloud vendors because clearly these same people know better than everyone else and these are actually simple problems if only we had their perspective and knowledge.

A 1000 times this. These are solved problems. Anything new has to be better to be taken seriously.

Re:Back to the '70s with Serverless by mlthoughts2018 on Sunday December 20, 2020 @01:15PM (#25488258)
Exactly, which points out how much better cloud tools are (given the level of seriousness most organizations are applying to them).
Re:Back to the '70s with Serverless by 1337shadow on Sunday December 20, 2020 @06:00AM (#25485398)
> The ego and arrogance of people in tech really shines through here.

You mean, "the ego and arrogance required to bend code and machines to their will and not the other way around". Maybe try to pass that and learn a thing or two.

Re:Back to the '70s with Serverless by kortilla on Sunday December 20, 2020 @03:41AM (#25484869)
Cool story, but the majority of that is not related to serverless. You still need a CI/CD pipeline with serverless and I’ve dealt with just as many miserable setups there.

> I can’t give the cloud providers my money fast enough

People run VMs and k8s on the cloud as well and run into every one of the problems you just described (aside from literal hardware failures).

Your rant seems to mainly be about having a good CICD pipeline and that’s not really related to cloud or serverless.

Re:Back to the '70s with Serverless by mlthoughts2018 on Sunday December 20, 2020 @12:53PM (#25488106)
I think you completely miss the point. With cloud tooling, my team is unblocked to spin up the resources we need, manage CI / CD in our unique way that solves our problems, quickly try out serverless if it’s a good model for a given use case, change logging or alerting, get new compute resources like GPUs, etc., all without being blocked by a choke point of in house engineering limitations and staffing limitations and political in-fighting over who owns what and who gets to define how abstract vs purpose-specific the tooling will be.

You toss out “build a good CI/CD pipeline” like that’s just some localized, solvable problem, and it’s exactly that lack of imagination and lack of understanding about the true multitude of use cases that causes this problem in the first place.

I agree “serverless” isn’t a panopoly solution to everything, but externalizing and democratizing all infra tools to cloud vendors is a major transformative solution that is worth a lot of money and solves a lot of problems.

For example, if someone like you had any leadership influence in my company’s infra org, I’d be trying to spend money on cloud tools just to get away from your authority as fast as I could, and I would be 1000% correct to feel that way and would end up with much healthier tool solutions and less blockages on use-case-specific delivery requirements by doing a giant spin move around your myopic perspective.

Re:Back to the '70s with Serverless by twh270 on Monday December 21, 2020 @09:08AM (#25495036)
This has nothing to do with technology or tooling, this is your team having the autonomy to do what works for you.

At my current client, my team is dependent on the "Platform Team" for _everything_ that isn't directly related to the code we're writing. Do we need a new Azure queue? That's a Platform ticket in JIRA. Do we need a new tool in our CI/CD pipeline? That's a Platform ticket in JIRA. Do we have a build failing for something other than tests? That's a Platform ticket in JIRA. Do we have an issue in production? We need to engage with Cloud Ops because we don't have access to anything in prod.

Plus, it takes six weeks to get a feature from development to production because horrible workflow and manual processes.

Plus, development teams are prevented from implementing anything without the approval of Solution Architects who are too far removed from the business to understand what's needed and too far removed from the technology to be able to design a working solution.

It's all fucking horrible, and yet we're using all the modern cloud tooling: Spring Boot, Azure, containers, K8S, Gitlab pipelines.

In contrast, a couple clients ago my team had full autonomy to do what we believed was the right solution. Stories would often be groomed, implemented, and delivered into production all within a 2-week sprint. No muss no fuss. When they weren't, it was normally due to us having to wait on another team to deliver their work.

Re:Back to the '70s with Serverless by FpUser on Sunday December 20, 2020 @07:50PM (#25491033)
>"...compared to the nightmare of shipping code on internally created developer tool & deployment platforms..."

I feel sorry that it worked for you this way (well rather it does not work). I have totally opposite experience. It takes single click of a button for me to build, test, deploy a code change using on premises and rented hardware.

I did cloud as well when it was mandated by client and it did not feel any simpler, faster etc. More complex and way more expensive it sure was but since it was not me paying the money I could safely ignore that part (not for my own projects and servers though as I do count my money).

Re:Back to the '70s with Serverless by 1337shadow on Sunday December 20, 2020 @05:47AM (#25485334)
I'm sure you will be able to setup proper pipelines on new projects within a 24h time frame and be brilliant at finding root causes by reading the stack trace and associated sources in the blink of an eye when you will be a senior hacker, I'm not worried about that!

Anyway, I think your comment proves that once we decide to invest in something, our brain does everything it can to justify that choice, Human nature really :)

Re:Back to the '70s with Serverless by mlthoughts2018 on Sunday December 20, 2020 @01:07PM (#25488201)
I don’t understand your comment. It seems like you think it carries a rhetorical punch to (I guess) suggest that in the cloud version we’d still be slow and make mistakes? The writing is so unclear it’s hard to tell. But if that is what you’re saying, I think you deeply missed the point.

Yes of course my team won’t solve bugs instantly just because of cloud tools and of course we will make mistakes implementing bad infra designs, especially early on as we are still gaining experience with efficient and cheap cloud patterns. Nobody said otherwise and my comment before has no connection to anything like that.

Rather, if we have control over our own infra, we can adaptively fix those issues for ourselves without getting delays and philosophical or gatekeeper arguments from central data center admins. We would have autonomy to change our own deployment system, provision more compute resources, generalize something to work with GPUs, test out different delivery and runtime frameworks, create our own ingress failure robustness patterns, all without blockages and mandated central infra bottle necks.

To create such a configurable and adaptable API-driven portal into bare metal or a data center yourself is way too slow and costly, only the largest companies can do it, and then they turned it around and made it into cloud vendor options.

When medium sized companies try to do it, the most common outcome is a ton of exceedingly arrogant SRE types will create piss poor tooling that barely works for maybe the top 2-3 use cases and then will whine and complain that they can’t support generalizing it for the long tail of other use cases, and because they have political power as the gorilla in the room sitting on top of the only interface to the data center, nobody can argue with them, and you get a burnout mill with constant turnover on all the other teams who have to do SRE’s job for them while listening to the arrogance about what a rip off cloud tools are.

Re:Back to the '70s with Serverless by 1337shadow on Sunday December 20, 2020 @03:19PM (#25489190)
I don’t understand your comment. It seems like you think "anyone talking about the rip off cloud tools" are "exceedingly arrogant SRE types creating piss poor tooling that barely works, whining and complaining that they can’t support generalizing it, sitting in actual data centers, with a burnout mill with constant turnover on all the other teams". But apart from that, I'm sure you're a lovely person to discuss with ;) Anyway, I was just saying that senior hackers have more options than you seem to have, also that we tend to find excuses to justify our investments once we are locked in them, it's perfectly natural and nothing to be ashamed of.