Our organization spent two years going through these gyrations for elastic search. They failed. It never happened. No other solution was brought on board, they simply couldn't justify the cost. All told they spend hundreds of thousands of dollars to come up empty-handed. Meanwhile my new team had a project where elastic search was the obvious component as part of the solution and we simply used that service on AWS. Done.
It's been a while since I've worked at a startup or a mid-cap company, but for any company in the Fortune 500? Cloud is a godsend!
But I like the idea of just doing it using cloud. Once a cloud vendor is approved, all its services are implicitly approved as well. Nice.
Exactly! When we start a new project the AWS Console is just a big tool box we open to see what's available that we can use to deliver the solution.
Either the company will have lost control of their spending and be sending 6-7 figure paychecks towards AWS. Or the ops team will have clamped down on the accounts and now you have to go trough the same process to get EC2 instances and other cloud resources.
Solving cultural problems with technology generally doesn't work.
Solving cultural problems is indeed difficult but most of us are not going to be at a company long enough to give a shit. The name of they game is taking money off the table and the best way to do that is to deliver value quickly and then move to the next table.
I pity the fool who hired you. Or will hire you, for that matter.
That's a ridiculous statement! Technology is the prime driver of culture! If you want to effect lasting change, the last thing you should do is try to convince people you are right. Set up the technical infrastructure to support the culture you want!
In fact, it's the reverse — culture has practically no impact on technology once deployed.
It does by getting rid of the people.
Very few organizations are able to stay efficient on the long run as they scale. This is a simple matter of processes being stacked on top of processes and inter-departement frictions. That complexity is inherent to having regulations and laws that you need to comply with that become hard requirements.
For a more concrete example think of HIPAA compliance, during an audit you can have everything on-prem and hope that your IT department actually understand the security and privacy boundaries or have everything in AWS and be compliant because they conform to the standard by default.
> syphon off any profitability
For most of those businesses software is not the profitable bit, it's a cost center like most other things that serve to support operations. In that context, migrating your code to AWS/GCP/Azure means very little cost inflation.
But at least it solves the basics.
So much of HIPAA is actually getting the administration right.
Simply using AWS doesn't do anything to help with this in that same long run. Give it a decade or two and those same organizations will suddenly have multiple bureaucratic departments to approve new AWS resources and spending.
All AWS is offering is a channel that a bureaucratic process hasn't been developed for _yet_ and the organizations hivemind response is to allow its usage freely because it doesn't fit in the existing form template of rules
A decade or two is someone else's problem.
And all that for less than the price of _one_ competent sysadmin's monthly salary.
How would you suggest a company would match that with local servers and sysadmins?
One sysadmin per 8 hour shift = 3 people for 24 hour rotation, you need 1-2 extra so that if one is sick the others don't need to pull a double. That's $400k a year alone.
Then you need to get the 12 identical servers purchased and shipped around the globe to colocation. Now at least one, preferably two, of the admins needs to be a proficient DBA so they can get the replicating SQL running.
Now you need to build monitoring for all of that and runbooks on what to do if crap breaks outside of office hours.
..and now a server has kicked the bucket, you need to buy a new one from $vendor, get it delivered and set up in Australia - you're in Michigan.
And in some years the servers are underpowered because your business has grown, now you need to buy 12 new ones and get them installed.
I can do all that from home without putting on my pants. I might need to grab a shirt if the AWS people want to hop on Zoom for something. =)
This is why we use the cloud and managed services, usually it's either out of our hands anyway or already fixed when we manage to log in. For the last two biggest downtimes I've encountered even Meta's crap was offline too (FB, Instagram at least).
Hey I can do that too in our on-prem environment because it's well managed. Hell, I can do that in my fricken homelab if you'll accept postgress.
Cloud is neat and the major vendors know how to manage it. That doesn't mean you need to pay them to use a well managed platform, but you do need to pay _somebody_.
Like a bootstrapped one or two person startup. Or could I borrow your globally distributed home lab? =)
The difference here is that I can pay AWS by the hour (or even by the minute or second). I can’t hire a sysadmin like that.
I can try a few high GPU instances for 6 hours and shut them down. But if I need to buy 6 4090’s and the hardware to run them, I’m stuck with the choice and down a few tens of thousands.
Oh hey, that's my cloud spend.
EOM I have nothing, every month.
Cloud is not always an obvious win, and depends a lot on where you work
BUT it depends on what field you're in. If you're managing CCPA/GDPR/HIPAA data it's a huge bureaucracy mess to get everything certified. Cloud operators usually have their documented and certified already, you just need to figure out your own part.
Come tell me how it went =)
I've yet to see a manager who wants to spend money up front for stuff they might possibly need in the future. YMMV
You would be spending that 20k regardless assuming the company is growing.
Good thinking!
I thought we figured out it's a shitty idea and moved past it?
Cloud computing can either be a necessity for a business, or a money-hole from which is derived little to no benefit to the company.
Most small businesses could thrive with a -local- small dedicated server/workstation, and locally managed network resources.
And, as a rule, gaming companies shouldn't build their own engines either. If you have the skills to do that, start selling the engine instead of the games you make with it. It's a much better business.
Define your responsibility model first, then your infrastructure consumption patterns.
If you can't answer those questions, you may be architecting yourself out of a job. I watched more than one company that was operating on razor-thin margins, depreciating hardware over a decade, move to the cloud and promptly bankrupt themselves.
If you're in a business where you've got an additional 20% margin to piss away on someone else's gear, your logic makes a ton of sense. If you're in one that's making 5% on every widget that goes out the door, you should probably be working with your management to figure out how to remove the inefficiencies from your process rather than just moving your problem onto someone else's hardware.
Obviously YMMV and the savings are going to be dependent on each project, but we've had enough solutions that we've had on-prem that we've shifted to AWS and show substantial cost savings that the business is comfortable with our taking new projects and just starting out in AWS.
what is the cheapest real actual server you can buy, new, with a support contract from a reputable vendor like HP/Dell/etc? (Assume we're running both the db and web-server on one server.) And what's the cost of sitting it in a reputable colo? And then multiply that by, let's say two, to get redundancy in a us-west and us-east region. Ideally, we'd want more regions to serve the entire world, but let's just go with two for now.
Let's say $10k for a server w/ support contract, x2, plus $400/mo for colo from Hurricane Electric (https://he.net/colocation.html) in us-west. It's a promo deal, and only in us-west, but let's just go with that price. $400*60 for 5 years (standard lifetime of computer equipment) = $24k. So $34k per server * 2 servers / 60 months ~= $1,130/month amortized. Thankfully, we're not in cloud, so that's the cost whether it's 1 request/second, or 1,000 requests/second.
A Raspberry Pi or old laptop in your basement with one consumer-grade ISP, and not factoring in the cost of electricity does not count here. Those are obviously going to be massively cheaper, but they're not remotely suitable for enterprise.
Meanwhile, Lambda gives you 1M requests and 3.2 million compute-seconds a month for free. This works out to be 0.4 rps (requests/second) or 22 requests / minute if spread out evenly. Which isn't, like, a lot, but you haven't paid a single cent to AWS for this yet. Let's say you want to get to 40 rps (to keep the math easy) or 100M requests over the month. According to the AWS calculator (https://calculator.aws/#/addService/Lambda), this'll run you $120/month, or $120*24 ~= $3k for 2 years.
But that doesn't count hosting for your static files with CloudFront, or does it count the RDS instance backing it, and we'd use AWS Cognito for user-auth since we've accepted AWS vendor lock-in. (Which isn't great, but that's the state of the industry.)
CloudFront: Free Tier 1TB, 10M requests/month. 10 TB/month will run you $1k/month https://calculator.aws/#/addService/CloudFront
RDS: This will vary greatly. A teeny tiny db.m6g.large instance w/ 2 CPUs and 8 GiB of RAM is going to run you $250/month, but a big one fat juicy db.r5d.24xlarge will run you closer to $20k/month. https://calculator.aws/#/addService/RDSMySQL
Cognito: With 1,000 active users, this'll be $50/month; 10,000 users is $500/mo, and 100,000 users is $5k/month.
Between Lambda, CloudFront, RDS, and Cognito you could easily blow past that $1,130/month estimate for a proper colo'd pair of servers, if your service gets at all popular. But if you stick within the free tier for Lambda and Cloudfront, then you're only looking at RDS and Cognito costs, which can vary greatly. Anywhere from $500/month to $26k/month, or more.
Except to get the same service in a colo as a $26k/month AWS bill, you'd need to spend way more and have much more than the single server in a rack that I started with. Thankfully, with something like Equinix Smart Hands, no one has to drive to the colo to futz with a server when it develops a bad hard drive.
Graph it out for your particular use case which one is better, but cloud infrastructure has the benefit of opex vs capex, as well as the opportunity cost of time. AWS Amplify will let you stand up a whole platform hacking on a Saturday before Dell can even quote you a price on a server or Equinix can answer the phone.
I also did reject a raspberry pi or old laptop server early on, but that's not to be underestimated. If I've already got some server running at home for my house, running a service on that, fronted by Cloudflare can take you quite far.
It really just depends on your use case.
All of the "and you don't get all these other services" is just a red herring because, again, we're talking about a fortune 500 that isn't building their services on a pair of servers. They buy racks at a time, at minimum, and unless they're grossly oversizing what they're buying, will have better performance per-dollar all-in, including the people cost and the surrounding infrastructure cost, to say nothing of the egress bandwidth because you know, their customers aren't just other AWS servers...
The major change in the last 10 years is that there's now a very stable & general open-source cloud platform that covers a much wider range of the stack than what we had before, and which has achieved a critical mass of popularity.
On prem is still hard. Having an organization that isn't crufty & slow, where there aren't forms in triplicate & long waits for every request, is still hard. But it's gotten a lot easier than it was to bring up, and to let users do some self service. There's a small scale org where having a small handful of clusters & letting users semi-free, mostly using ci/cd, is quite attainable. Figuring out how to scale up to serving a mid or larger size org is still quite a challenge.
However, if you have I do worry that once people have fully migrated to the cloud they will have a captive audience and will start ramping up the costs.
This suggests that the way corporations do IT is radically wrong and actively harmful.
Companies utilize HR services, payroll services, and legal services, amongst many other services, because that's not in their wheelhouse and a part of their core competency, so why wouldn't they utilize cloud platforms as well? Like I said, managing and running multiple data centers having high-availability and DR capability isn't easy, or cheap.
The technical people want to solve technical problems and don't give a fuck about the company mission. The immediate management wants to improve the measurable recurrent tasks and don't even think about supporting anything different. The top management wants to cut jobs, minimize the need for competency, cut costs, minimize flexibility and lean time to think about non-recurrent tasks because IT is a cost center.
The result is that all of those problems you point get solved. That's not the issue. But nothing of value is allowed to exist.
This seems like a wound self-inflicted by a company that can't seem to run itself efficiently. You're not describing attributes of on-prem and cloud--you're describing a dysfunctional organization.
If you've never done the work or been around it, the difficulty becomes a bit mythologized.
It’s very easy to run something with several 8’s of uptime if you’re not careful.
AWS has good patterns to guide you into the pit of success.
Yes, really.
My one irritation with stuff like AWS is that a lot of it is needlessly esoteric, to the point where I think they make their docs bad on purpose to get you to buy their enterprise support packages.
I was hired at Amazon 8 years ago to implement a couple of features. One of them was difficult because it required communication / buy-in from multiple two pizza teams. By the time I left it was stalled. Three years ago I randomly started at an AWS customer who needed that feature as they had followed the AWS conventional wisdom for setting things up and architected themselves into a corner. I was able to craft some workarounds but they're hacky.
AWS has been promising us this feature "next quarter" for the last 3 years.
If you look around, you can find a number of weird corner-cases where AWS does not make it easy to do the simple thing you want to do, but you can hack around it.
We went looking at GCP and Azure and there are similar issues, though in different parts of the infrastructure.
Not to mention each of them change their interface frequently enough to be annoying. Thankfully, reverse incompatible changes are not at all the norm.
My cloud infrastructure is replete with architectural warts to avoid quirks we didn't know existed until after we deployed. My on-prim components, though managed by a different team, do what we want them to do because the team who built them asked us what we wanted before building it.
AWS weirdness accounts for about 85% of my time. On prim weirdness for 5%. But yes, local-first is obviously a non-starter.
I’ve hit multiple oddities with cognito (only supporting custom claims in id token), lambda (consuming SQS messages without invoking the function when run in low concurrency).
So there are strange edge cases, today RDS and ECS are the services of choice, I wonder where I’ll be in 12 months.
AWS IAM is baked into every single product natively. It isn't perfect and their JSON dialect is annoying at times, but having granular RBAC for storage, compute, ops, network in a single language is incredible for security.
And using IaC, you can put guardrails on specific tasks that IT does often. Manual reviews become automated.
It is a ton of conversion and up front work, but there are upsides.
And then of course there is the instant global reliability, where a lot of formerly complicated sysops becomes automated as well
Final thought: other than the hardware abstraction, everything I talked about re: IAM could be done with a local software stack, if it existed.
you do realize that "cloud guardrails" often started out life as "hacky bespoke solutions". you are assuming more business risk than is necessary.
It's like taking an ops support team that is using perl, java, php, python, bash, ksh running on RHEL5 and HP-UX and getting everyone on RHEL8, terraform and Go.
So for the first few months, it was moving all business logic to lambdas/RDS then the next year, moving it all back elsewhere - not necessarily on-prem, but anywhere except AWS.
So I'd expect your cloud team to get very much like infra once cost realities become involved. It's possible your volumes/costs are much less for the features you build - in that case, kudos to designing stuff that is perf-aware - but sometimes you don't control volume and that's a cost-risk.
> Anyone who's worked inside a large organization knows the cloud is the solution to be able to get things done and deliver value to the business.
Local first is focused on end consumer experiences and is orthogonal to the on-prem vs cloud argument; ideally there wouldn't even be a server. @pvh and i&s team have done a great job threading the needle in articulating the ideology. At a high level it is advocating for products that can provide the benefits of the cloud without being locked on or turning your phone into a paper weight if it doesn't have internet access.
Nah. Multiple F500s to include manufacturing, railroad, and aircraft. Pick an industry, been around it.
3 for 3 -- large cloud rollouts clobbered the IT team and blew OpEx out of the water. Saw VPs catch an axe for it, and at least one CTO get nailed with embezzlement accusations due to a "cloud consulting" company he had ties to, and who managed to get a no-bid contract for north of 400k.
Never been anywhere that it wasn't a kind-of-clusterfuck, and the only answers in reply I've seen were No True Scotsman-type "you're just not embracing the cloud and it's changes to processes bla bla bla".
Why?
On-prem solutions require I work with the infrastructure group, who wants to cookie-cutter me into their limited stable of infrastructure options; I have to work with the systems group, who gives me very limited options and configurations for the operating systems; the network group, who are very fixed in their ways and make very few concessions; and the middleware group which is what you need to connect to anything else of importance. Four groups! All with their own workflows, their own SLAs, their own exemption policies, etc. It's a nightmare to get anything done and the business is frustrated that things take so long.
Now I'm on a team that does everything in AWS. We keep everything serverless so the infrastructure and systems teams can't butt in, and the cloud environment eliminates the need for interacting with the network group. I only have the middleware group to contend with. The result? We're delivering much-needed solutions far faster than any other group in the organization. They also have far fewer business impacts (read: support issues impacting operations) than the applications we have on-prem.
People seriously underestimate the time and energy required to keep a highly-available, and DR capable datacenter running. They also underestimate the complications of interacting with all the teams need to keep that datacenter running. We have found that AWS is faster, cheaper, better than hosting on-prem.