There's also this enduring myth you see repeated even here of how much saving there is by not having "devops", when only very big companies needed them in the first place. A few minutes per month is the most devops any startup would need to do in reality running their own VMs or servers, after an hour or two of setup. And that hour or two is often less than faffing around getting all the labyrinth moving parts of AWS/Azure/etc. working.
And the whole thing becomes a transparent farce when you look at providers like Azure, where their own admin interface is so slow and prone to random breaks you have to ask:
How is this scalable?
It's clearly not scalable for themselves, so how is it "scalable" for everyone else?
There are benefits, mainly for me I'll use them for hassle free deploys[1], but the whole thing sometimes feels like a massive con to sell the performance of a potato by advertising it as a jet engine.
[1] Like a month back I deployed a demo prototype web app I wrote in 3 hours onto Azure straight from github with like two clicks, and got the contract, can't really beat that.
I dunno, the first company I worked at was very small, under 30 people, sales included, and I was hired on as an SA. If they were using serverless, they would not have needed to hire me.
Deployments and system upgrades were very toilsome. The dev lead would spend a few hours deploying new code and we’d only do it around midnight in case there was a problem (our customers were almost entirely US based). I spent a lot of my time automating OS upgrades on the web servers behind our proxy. All of this work would be unnecessary for that small company if they were using serverless.
Additionally, as mentioned before, the majority of their traffic was during the day so they could have scaled down automatically at night, cutting costs. Would this be cheaper than running machines 24/7? I’m not sure, but I could see why you’d be interested in at least looking into it
Not using serverless doesn’t mean there can’t be a CICD pipeline.
Don’t get me wrong—serverless platforms are not a ci-cd silver bullet. Many organizations build pipelines on top of serverless deployment apis, but it certainly does take care of a lot of things for you—not to mention other features mentioned elsewhere in my comments
And since we have containers it's become really much easier actually ... Deploy a Traefik container which will watch dockerd socket and self-configure HTTPS and all when your other script starts container. Really, it's much easier than back in the days: https://github.com/jpic/bashworks/tree/master/vps
[0] https://aws.amazon.com/certification/certified-devops-engine... [1] https://en.wikipedia.org/wiki/Idiocracy#Plot
My reality at a 20 person shop with a few devs is much closer to the OP - a few hours setup (we save the cloud config and can remake servers in a few minutes), servers run for months without intervention, uptime is on par with cloud services, deploys take seconds. Managing servers or deploys just are not major problems for us.
Serverless is not a solution to deploy problems - mostly because they’re just not significant problems for us, if anything the dev deploy story is worse than renting VMs. It’s an interesting take on serving requests and I do believe some variant of it might be the future (a variant more like cloudflare workers IMO), but at present there are serious drawbacks and it is not clearly better, particularly when provided by a predatory company like Amazon.
I remember using companies like MediaTemplate and FullControl 10 years ago for clients and not having to worry about OS updates.
Right up until something crashes or some hardware fails.
Unlike the public cloud where Nothing Ever Goes Wrong (tm).
There are never brownouts.
There are never latency spikes.
No undocumented quota limits.
Documented but confusing quota limits.
Unexpected performance issues caused by other customers.
Forced upgrades.
Forced patching.
Shock bills.
No siree Bob, that kind of thing just doesn't go on in the public cloud! It's all rainbows and unicorns....
Your original claim is quite unrealistic and you'd do well to retract it. It's bad advice both for public cloud and for self-managed efforts alike.
We supported disaster recovery and high availability deployments.
Any failed component, Dell would send us a replacement in less than 24 hours.
Try calling Amazon and getting that kind of response for your problems.
But, yeah, that is worse-case scenario.
Sure serverless etc has it's upsides and in that case I do use it, I just encounter close to 0 companies per year that need it or benefit from it. Aws ways benefits.
The original claim is not at all unrealistic, I’ve been running a small shop with about 20 VMs without issues for years, uptime seems to me about comparable with serverless or hosted services from big players who regularly have small outages and there really is very little work involved.
It exists to shift CapEx to OpEx for accounting. Any technical considerations are a distant third place on the list.
First, the markup in just sheer ops terms is enormous. Minimum 4x, and in some cases 20x. Sure, you get a couple more nines of reliability, but 99% of startups don't need that much reliability.
Second, in countries like Canada where capital expenditure on computer systems can be heavily marked down[0] people still use AWS and DigitalOcean by default.
The main reason people don't do colocating is that it is a waste of time better spent. DigitalOcean is cheap enough. AWS is cheap enough and if the team doesn't have experience running Postgres, fine, they use RDS. Sometimes it's just sheer laziness at first that later blossomed into "holy shit, we're growing too fast to even think about migrating off of these giant AWS bills" which was essentially Dropbox's story if I recall correctly.
[0] 55% per year, and in practice they allow full expensing in the first year for petty amounts of outlay. For example, if your business is a data centre, sure it's 55% per year, but if you run a flower farm and fully expense a new laptop for sales the CRA is generally fine with it.
(Of course you have to force your devs to do the ops stuff now, but that's not accounting and HR's problem now.)
No technical reason, maybe. There is a social reason though: if you point out that there is no need to build scale (yet), you are pointing out that the business will not achieve overnight success. Even if it's true, it kills excitement and is therefore not something that people want to hear. Who wants to realize that what they're spending large amounts of time on (and possibly even neglecting other things like health, sleep and social contacts) will not even make all their financial dreams come true? Much easier to not think too hard about it and keep on doing what everyone is doing.
So, building out scalable infra is signaling that you too believe in the company and are preparing for the moment (any time now!) that "webscale" is needed. It's showing that you are part of the team.
Serverless, like every design decision, comes with its own set of trade-offs. Whether those trade-offs make sense for your environment is something you have to determine. At my company we have seen a productivity increase and shorter development times using serverless. Of course, that was after the learning curve which is certainly not trivial.
Apparently, you don't even know what that word means. You can not "have" DevOps. It's a methodology and not a thing. Besides, DevOps means exactly the opposite of what you are implying: it is the practice of not having a dedicated operations team and letting devs handle as much of the operations for their own applications as they can.
> few minutes per month
I can tell you my team spends a good 30% of its time on operations. Some of that (for example adminstrating K8S clusters) could be offloaded to cloud providers for money.
Yes, a technology probably won’t live up to its marketing.
Yes, what works is probably good enough.
Yes, changing has switching costs and sacrifices institutional knowledge.
Yes, we’ve seen these technologies before and switched away from them from various reasons.
That’s all fine and sometimes frustrating as an engineer. But it’s not the only reason decisions are made.
Sometimes pitching your exec team on The New Hotness gives cover for changes that actually do need to happen.
Sometimes adopting The New Hotness is marketing to recruit engineers who want to try something new.
Sometimes adopting The New Hotness is because some people are bored and uninspired in their jobs and just want to believe in something good again.
The success of things like MongoDB and blockchain are not because they are better or even very good at most things people use them for. They’ll never live up to the hype. But people need things to get excited about. It’s psychological. Attacking them purely on technical points can lead one to miss the actual point.
I wrote a blog post on Monday about a relatively old and not-hot language. The post nevertheless trended on Medium and received 50k views. A lot of the feedback was from people who were just happy to see someone senior say something positive about the not-hot language.
Anyway, working with The New Hotness also gives people something to put on their resumé: “I built system X on top of The New Hotness. Nobody had built such a system before for The New Hotness”.
So people keep moving on to the next project, leaving the old ones running live but to rot, and at best every once in a while someone will be assigned the dreadful task of having to figure out why the "legacy" (even though it's still fully in operation and servicing customers) system had an outage and can't recover.
Our field suffers from a serious lack of focus. People don't learn to work under constraints. Instead, they prefer to build a whole new thing, full of complexity and hidden issues, just to get around a few nuisances that, with the right mindset, actually force one to be creative (which should make work fun).
I've recently commented here on HN about how I feel like I'm outside the norm on these things [0]. I was happy to learn I'm not alone. I do think those of us who don't mind working on "legacy" systems and perhaps with limited tools are the minority though.
Sometimes it's not even about working on old systems. You work with current tech, build something, and as soon as it's barely deployable, you're expected to move on to the next thing. "It's a great opportunity for your career," they say, "you're gonna get to build something new and make key decisions."
Why does it have to be decided for me? It's my current predicament. I'm working on a project that took me two months to get in the right mindset for and get all the context I needed. I had to try things out, sort out ideas, until I got a clear picture of what to build. Now I've built the first iteration of it. It works, and I'm super excited about it. But it's not done by any means. It needs more features. It needs to be polished so it doesn't make operations miserable running the thing. And yet... I have to move on, and hand it over to someone who doesn't have the same excitement for it that I have, and has maybe 10% of the context.
All because building new things (and adopting The New Hotness, as you say) is what "everyone" wants.
I've chosen to take on more than one heavy maintenance job, no client facing visibility in terms of fancy new features. Feedback in performance reviews would always be "it's hard for people to see what you're doing." In the end you have to find your own way to game the system to get appropriately recognised, but of course that kind of office politics isn't really what you want to spend time either.
I'm doing more 'new feature' style work now but I'm happy to not be tumbling through the waves of hype. We get a little bit of new hotness, because of course you need to embrace some aspect of that to avoid stagnating and getting too stuck in your ways, but not so much of it that the engineering team is incentivised to churn through half-assed features.
Managing this stuff is a job, and it requires discipline and organisation. It's one of the few functions of an IT department that is left once you go to cloud (as dealing with Oracle's bullshit is then gone). The younglin's doing development in the business functions have to play by the rules - and the business functions need to agree that there are rules or they will :
a) spend much monies on the issue you describe
b) get cracked open like fresh eggs by any passing hacker who is keen on sports cars
The problem we have in software is that we have engineers build bridges and then somehow expect them to stay around to paint the thing, change light bulbs, etc and stay fulfilled. A person who is an expert at making new things won’t like being in maintenance. It’s that simple.
It's much more fun to improve the status quo by inventing something nice as to repeat the work of the past generation (only painting things in a different color).
Even without cloud APIs and containers and whatever, I find there's a creeping thing that happens here which has been helpful to become consciously aware of and intentional about fighting back on.
Basically, the thing is that wrappers become interfaces. You write tool A, and then tool B, and then some kind of script C which calls the two tools and produces a result. Then someone says "dang, so convenient", and sets up a Jenkins/Rundeck/Airflow/whatever job that runs script C against some source of inputs automatically— maybe there's even a parameterized build capability where arbitrary inputs may be submitted to it on demand.
Before you know it, the fact that the original tools and script exist has been forgotten, because everyone's interface to this functionality is a handy webpage. And when someone wants to add more features, they don't even go back and change the script, they end up just adding it directly into the job definition. Before you know it, the full scope of what is being done on "the system" isn't even really available any more to be installed, invoked, and tinkered with locally, not in a meaningful way, anyway.
It's not even necessarily all bad, but when I see this happening in a domain where I have the awareness of the underlying tools (or I may even have written or contributed to them), I do try to help demystify them by writing up docs on how to do the local setup, helping others get going with it, etc.
"Serverless" always reminds me of CGI/fastcgi, which was still a thing the last time I wrote web apps for money. It had the advantage of "scaling to zero" and could run a huge number of low-usage apps on a single 2000-era server.
I really don't know why that is, my guess would be that few people have really built something beginning to end with the most boring tech possible (start out with PHP+MySQL like everyone did 15 years ago). Or they always operate at levels of abstraction where everything is slow, so they have simply gotten used to it, like their text-editor not running smooth unless it's on an i9, because the text-editor now is a pile of abstractions running on top of Electron when vim was able to run smooth decades ago. It's sad and both an opportunity at the same time, because you can be the one with the correct gut feeling of "This should really be doable with a single machine, if not we're doing something fundamentally wrong".
We have the computing power to run complex physical simulations or AI training sequences on a normal home computer, but for some reason use programs that take 100 times longer to start than old software despite not having more features, and websites that sometimes even come with loading screens. Electron isn't even as bad or slow as many people think, but somehow developers still manage to throttle it so hard that I might as well just use a website instead.
As someone who is just starting with nodejs and web development I find a lot of the tech feels nice but sometimes also unnecessarily abstract. Sure, it tends to make a lot of the code very elegant and simple, but every additional framework adds another guide and documentation to look at, another config file that has to be correctly referenced by an already existing one so npm knows what to do, another couple of seconds of build time and another source of bugs and build problems. Then of course you need that trendy package for easier imports maintenance - which IDEs automatically handled in the past, but now we gotta use an editor running in a hidden web browser which started from scratch in terms of features but is just as slow in return.
Going through my current project has really worn me out with regard to tolerating UX slowdowns in developer tools. I am getting so incredibly tired of how shitty visual studio performs on a threadripper with effectively infinity ram. I've had to turn off all of the nice features to get our codebase to be even remotely tolerable. And, dont get me wrong. This is not a plea to Microsoft to fix VS. I think everyone involved can agree that fundamentally VS will be unfixable WRT delivering what I would consider an "ultra-low" latency user experience (mouse/keyboard inputs show on display in <5ms). Any "fixing" is basically a complete rewrite with this new objective held in the highest regard.
Current developer experiences are like bad ergonomics for your brain. This madness needs to end. Our computers are more than capable. If you don't believe me, go play Overwatch or CS:GO. At least 2 independently-wealthy development organizations have figured it out. Why stop there?
That's not possible, at least if you measure the full time from actuating the key to the screen showing the updated content. The record holder in that regard is still the Apple II with 30ms[0].
But I agree, modern software should react much faster. It's kind of weird how on one hand we have hardware that can run VR with 120Hz and extremely low input latency, and at the same time some programs that only need to display and change a couple bytes in a text file need 4 seconds to even start.
That's a lot of it, I believe. One startup I dealt with was honestly proud of how they were scaling up to hundreds of requests per second with only a few dozen AWS VMs to handle it.
I'm old enough to know how ludicrous those numbers are, having built systems handling more load on a single desktop machine with 90s hardware.
But I've come to realize some of the younger developers today have actually never built any server implementation outside of AWS! So the perspective isn't there.
I cured this one when my native server running on not so expensive multicore CPU with enough RAM left their cloudy setup in a dust performing more than a 100 times faster and still having huge potential reserve in vertical scalability. They were just astonished at what a single piece of modern hardware in combination with the native code could do.
When the server gets even remotely suspected of having been hacked into, you must trash it overnight and never use any of it again.
Sure, if a drive breaks you'd probably want to replace it instead of just replacing the entire machine; it's a thing you don't need to do with VMs, but that doesn't mean it's difficult. It definitely doesn't take "hours".
Managing physical setups is different from managing fleets of cloud instances, but it's not necessarily automatically inferior.
It is pure scaremongering and making not very nice assumptions about people's abilities without any real substance. If this is the idea of attracting customers or advocating the approach it seems like a very poor job.
Anyways nice job advertising cloud and trying to scare people into using it. You happy with it - good for you. No need to convince me as I have my own reasons and do not have to buy into yours.
I'm about halfway through my career and I have seen something like that happen exactly once; the erratic behavior of the machine had my team scratching our heads for a fair bit until we ran memtest86 on it after running out of ideas. So I would say that it's not impossible, just extraordinarily unlikely.
The rest of your post I entirely agree with.
One word: Worldcom
I still remember coming up to 2000 sitting in a meeting room of an equipment provider I recently joined looking at projections of market growth, competitors and our own sales and profitability. Shortly afterwards that part of the business was sold, I remained with the other non internet part and we sailed through the .com crash unscathed.
A lot of what AWS offers makes no sense. A lot what some AWS customer's teams are building is overly complex and a horror show to maintain. But there is also value: Maintained images, backups, on demand resources and specialized resources. We hit one of the inflection points in Moore's Law and that drives us into horizontal scaling and specialized hardware.
Like with Worldcom and the fantastic internet growth story back then there is a kernel of truth and value. Find people that understand it and be glad others are playing around and subsidizing the investments in the core.
This is every company I’ve ever worked for or ever heard any colleagues, friends or associates work for, that tries to build their own SRE practice on top of data centers.
Large cloud platforms alleviate so so much of this pain. The “back to the 70s” workflow sarcastically described in the quoted passage you shared sounds like an absolute dream compared to the nightmare of shipping code on internally created developer tool & deployment platforms interfacing to bare metal or space leased from data centers.
I can’t give the cloud providers my money fast enough and I have only ever felt happy about that for years and years now.
Are all the commenters in this thread just cynical engineers who really can't see past their own narrow world view? I get snake oil is snake oil, but there is such a strong brand of "anything new must be snake oil" that runs through the HN community it boggles my mind.
The world is actually full of complex systems. And only once these systems are in place does the software engineer get to do his work! People on this site act as if they have all the understanding and knowledgeable necessary to decry something like cloud vendors because clearly these same people know better than everyone else and these are actually simple problems if only we had their perspective and knowledge.
The ego and arrogance of people in tech really shines through here.
I would say there's at least as strong a "brand" of "if it is a new way it is probably a better way" on HN as well.
I think making blanket assumptions in either direction is a terrible idea. Best to be aware of your own biases, and try to be as objective as you can in your decisions.
Experience matters, and there are new solutions that are inferior to older ones in many cases (yes, there is plenty of "snake oil" for sale). However, sometimes new people look at an old problem with fresh eyes and come up with better ideas on how to solve it.
This really isn't anything new. When I was fresh out of college I remember older programmers talking about "such-and-such" is just the old way of doing it repackaged. There is lots of oscillation in the computer industry.
A 1000 times this. These are solved problems. Anything new has to be better to be taken seriously.
You mean, "the ego and arrogance required to bend code and machines to their will and not the other way around". Maybe try to pass that and learn a thing or two.
> I can’t give the cloud providers my money fast enough
People run VMs and k8s on the cloud as well and run into every one of the problems you just described (aside from literal hardware failures).
Your rant seems to mainly be about having a good CICD pipeline and that’s not really related to cloud or serverless.
You toss out “build a good CI/CD pipeline” like that’s just some localized, solvable problem, and it’s exactly that lack of imagination and lack of understanding about the true multitude of use cases that causes this problem in the first place.
I agree “serverless” isn’t a panopoly solution to everything, but externalizing and democratizing all infra tools to cloud vendors is a major transformative solution that is worth a lot of money and solves a lot of problems.
For example, if someone like you had any leadership influence in my company’s infra org, I’d be trying to spend money on cloud tools just to get away from your authority as fast as I could, and I would be 1000% correct to feel that way and would end up with much healthier tool solutions and less blockages on use-case-specific delivery requirements by doing a giant spin move around your myopic perspective.
At my current client, my team is dependent on the "Platform Team" for _everything_ that isn't directly related to the code we're writing. Do we need a new Azure queue? That's a Platform ticket in JIRA. Do we need a new tool in our CI/CD pipeline? That's a Platform ticket in JIRA. Do we have a build failing for something other than tests? That's a Platform ticket in JIRA. Do we have an issue in production? We need to engage with Cloud Ops because we don't have access to anything in prod.
Plus, it takes six weeks to get a feature from development to production because horrible workflow and manual processes.
Plus, development teams are prevented from implementing anything without the approval of Solution Architects who are too far removed from the business to understand what's needed and too far removed from the technology to be able to design a working solution.
It's all fucking horrible, and yet we're using all the modern cloud tooling: Spring Boot, Azure, containers, K8S, Gitlab pipelines.
In contrast, a couple clients ago my team had full autonomy to do what we believed was the right solution. Stories would often be groomed, implemented, and delivered into production all within a 2-week sprint. No muss no fuss. When they weren't, it was normally due to us having to wait on another team to deliver their work.
I feel sorry that it worked for you this way (well rather it does not work). I have totally opposite experience. It takes single click of a button for me to build, test, deploy a code change using on premises and rented hardware.
I did cloud as well when it was mandated by client and it did not feel any simpler, faster etc. More complex and way more expensive it sure was but since it was not me paying the money I could safely ignore that part (not for my own projects and servers though as I do count my money).
Anyway, I think your comment proves that once we decide to invest in something, our brain does everything it can to justify that choice, Human nature really :)
Yes of course my team won’t solve bugs instantly just because of cloud tools and of course we will make mistakes implementing bad infra designs, especially early on as we are still gaining experience with efficient and cheap cloud patterns. Nobody said otherwise and my comment before has no connection to anything like that.
Rather, if we have control over our own infra, we can adaptively fix those issues for ourselves without getting delays and philosophical or gatekeeper arguments from central data center admins. We would have autonomy to change our own deployment system, provision more compute resources, generalize something to work with GPUs, test out different delivery and runtime frameworks, create our own ingress failure robustness patterns, all without blockages and mandated central infra bottle necks.
To create such a configurable and adaptable API-driven portal into bare metal or a data center yourself is way too slow and costly, only the largest companies can do it, and then they turned it around and made it into cloud vendor options.
When medium sized companies try to do it, the most common outcome is a ton of exceedingly arrogant SRE types will create piss poor tooling that barely works for maybe the top 2-3 use cases and then will whine and complain that they can’t support generalizing it for the long tail of other use cases, and because they have political power as the gorilla in the room sitting on top of the only interface to the data center, nobody can argue with them, and you get a burnout mill with constant turnover on all the other teams who have to do SRE’s job for them while listening to the arrogance about what a rip off cloud tools are.
I've mentioned as much to several C-suite executives and professional venture investors I know who tend to wax philosophical about "exponential growth in cloud services and big data" and whatnot. Invariably, their response has been to tell me that I just don't get it, or that I make no sense. They give me puzzled looks and maybe even get a bit exasperated with me.
Surely others here on HN have run into similarly minded individuals too. They're not hard to find.
Plus ça change, plus c'est la même chose.