If you have petabytes of data in AWS you already have a number of on-staff engineers with significant six figure salaries.

If the problem is that your group of six-figure salary people only know how to put data into AWS, or other cloud services, and not design/engineer/maintain your own bare metal infrastructure as well, then that would definitely be a limitation.

For reference, a few petabytes of data is not actually that many systems these days, if you have something like a bunch of 72-drive supermicros or equivalent with 14-16TB drives in them. Set up properly this can be administered by one FTE (of course with additional staffing/tech resources for when that FTE is on vacation/unavailable, and appropriate training for other persons who might have admin on the setup).

my very rough calculation here says that a 36-drive ZFS RAIDZ2 composed of 16TB drives is something like 492TB (447TiB) usable storage capacity.

so five such arrays would be 2460TB.

compared to the monthly AWS bill for 2.0 to 2.5TB of data you could probably afford to entirely duplicate the whole setup in a twin identical set of hardware at a geographically diverse off-site location.

> If the problem is that your group of six-figure salary people only know how to put data into AWS, or other cloud services, and not design/engineer/maintain your own bare metal infrastructure as well, then that would definitely be a limitation.

It's not about whether or not the engineers can make the colocated setup work.

It's that you're going to pay a lot of hidden costs with a colocated setup. Engineers can't set up, maintain, and do on-call for the colocated setup without subtracting from their primary working hours.

Each additional engineer you have to hire to help with the colocated setup is $200-400K fully loaded out of your company's budget. If you have to hire 3 additional engineers to fill out your colocated on-call schedule and help set up and maintain the system, that's easily an extra $1 million per year on your budget. Cloud is expensive, but $1 million goes a long way.

It's easy to look at a potential AWS bill and a potential colocation and hardware bill and declare colocation the winner, but then you still have to set up and maintain it all as well as constantly train everyone on it.

With AWS, you can hire engineers with AWS experience and they'll understand the big picture of how to work with things on day 1. With a custom setup, you're at the whims of whichever employees set up the system because they know it best.

Colocated systems tend to work very well at first when the original engineers who set it up are all still at the company and it hasn't run long enough to start encountering rare failure modes. They quickly become a nightmare when your engineering staff turns over multiple times and nobody can remember who knows how to do what on the colocated system or if the documentation is up to date or not.

> Colocated systems tend to work very well at first when the original engineers who set it up are all still at the company and it hasn't run long enough to start encountering rare failure modes. They quickly become a nightmare when your engineering staff turns over multiple times and nobody can remember who knows how to do what on the colocated system or if the documentation is up to date or not.

Everything above really sounds like it's just regurgitating AWS sales person talking points.

Sounds like a systemic management / CTO-level problem to me if a company isn't willing to put in place the hiring practices and compensation, documentation systems and operational procedures to deal with that sort of concern.

If your core engineering staff is turning over multiple times for arbitrary reasons you have other problems to deal with.

> Engineers can't set up, maintain, and do on-call for the colocated setup without subtracting from their primary working hours.

If a company can't hire datacenter techs to install hardware, cables, and swap hardware as smart remote hands, maintaining as little as a couple of 45RU cabinets of gear, you also have other management/systemic problems to deal with. I'm looking at this from the point of view of a facilities based bare metal ISP that owns/runs all of its own hardware, and can tell you it's not rocket science.

> If you core engineering staff is turning over multiple times for arbitrary reasons you have other problems to deal with.

People leave for all sorts of reasons: Moving for family reasons, becoming stay-at-home parents, moving for a spouse's job, retiring, starting their own companies, or even just getting bored and wanting to do something different. Or it could be as simple as getting promoted to a different role.

It's unrealistic to make engineering decisions with the assumption that the same engineers will be around and stuck on the same project forever.

Like the OP said: Every hour they have to spend working on the colocation setup is an hour they aren't spending on your company's competitive advantage, so you have to hire more engineers (and more managers) to compensate.

> If a company can't hire datacenter techs...

How many techs do you think you need for reasonable on-call coverage? 3? 6? Add a manager in the mix because you need someone to manage them.

The costs add up quickly.

It's weird to see people championing colocation as a cost saver and then pivoting to arguments that you just need to hire more engineers and techs and manage them.

Employees are expensive. One of the primary benefits of cloud is that you don't have to hire and manage all of these employees to do all of these things at the colo.

My team of three looks after hundreds of bits of diverse kit in dozens of locations across the world.

I can’t remember the last time I took a call outside of office hours, and even in hours it’s very rare. There’s enough resilience built in that any issues can wait until morning

The last major outage was in 2017, before we had a third member of the team. I was on the other side of the world installing a new system, the other was on leave. We had a network issue, OSPF melted and knocked out some services, we were down for about half an hour as I rebooted the core switch pair remotely.

(We’ve since redesigned so that doesn’t happen)

We get paid nowhere near six figures either.

Sure you can be ridiculous, I remember one team I worked on that employed a full time unix contractor (on 3 times the staff wage) to look after 6 servers and deploy a tar ball every few months. I replaced him with a small shell script. Another was a DBA looking after a small oracle database (oracle - which of course is that generations “just use amazon”)

I think a number of people who are taking the position of "but it's so HARD, and so EXPENSIVE!" to own and run bare metal network infrastructure may not have ever seen a proper OOB management setup, with dedicated OOB network, serial consoles on stuff, management routers and switch at site, things like cradlepoint LTE radios stuck to the top of colocation cages, etc.

And then basic other things like having remote smart hands ready to go, and common failure items like fans, power supplies, fan trays, hard drives pre-positioned and ready to swap in. With MOPs for swapping them. Stocks of basic things like fiber patch cables, commonly used transceivers, copper patch cables, stored in every cage.

> I think a number of people who are taking the position of "but it's so HARD, and so EXPENSIVE!" to own and run bare metal network infrastructure may not have ever seen a proper OOB management setup, with dedicated OOB network, serial consoles on stuff, management routers and switch at site, things like cradlepoint LTE radios stuck to the top of colocation cages, etc.

Or we have seen all of this and that's exactly why we don't want it.

Building a company is hard enough. Adding the overhead of developing, maintaining, recruiting for, and staffing our own datacenter is madness when I can click a few buttons and get the same thing from a cloud provider without hiring anyone extra to manage the datacenter.

No one is denying that a proper data center management system can exist. We all know it can exist.

The issue is that it's a huge distraction with a lot of potential pitfalls. Your network infrastructure with Cradlepoint LTE radios in the colocation cages sounds great after it works, gets set up, stays documented, and all the bugs have been ironed out. But that's a lot of hidden work that could have been allocated to launching the product faster.

I think the difference in point of view here, is that from my own perspective, owning and running the bare metal things is a basic core competency of being an ISP. Which is what I do for a living. The infrastructure I've described up thread is the product.

If the use case is somebody developing a software product that is a totally other scenario.

For an ISP, you are probably correct.

> Building a company is hard enough. Adding the overhead of developing, maintaining, recruiting for, and staffing our own datacenter is madness when I can click a few buttons and get the same thing from a cloud provider without hiring anyone extra to manage the datacenter.

“Building a company is hard enough, adding the overhead of actually building a company is madness”

> But that's a lot of hidden work that could have been allocated to launching the product faster.

The topic of this thread isn't about a brand new start-up with no resources trying to build their product as much as possible, it's about a company that spends millions in cloud invoices.

Building a company using AWS, totally legit. Scaling this usage up to millions of dollars because you don't want to manage the hassle, fair enough it's your money, I don't mind if all your profit goes to Amazon really…

^^^ 1000x

One big pain that no one seems to mention are the mundane details that come with running your own DC. Even in our fairly small operation we had half of a person dedicated to maintaining the warranties on the hardware, sourcing replacement hard drives for 10yo servers, disposing of retired hardware, purchasing new hardware, etc.

These things take a huge amount of focus and don't contribute to your product at all. The "cloud" has removed all of this busywork and I wouldn't go back to the old way for almost any amount of money.

Been there, done that. No thanks. Every place that has their own hardware also has a huge bureaucratic process to get more hardware for your project. Not to mention almost always the software stack is old as dirt. For example mongo might be two or three major versions behind current… and IT wants nothing to do with supporting the new version.

People move to the cloud to escape their company’s IT process… there might be some unicorn company out there that does infrastructure “right” but I’ve yet to work there.

^^ This 1000x.

Humans are incredibly expensive and notoriously unreliable when compared against “machines”; or in this case an API.

It’s usually worth paying 2-3x the cost to have someone else manage something for you with a given SLA, because that’s what it will end up being when you decide to bring it in house when you take into account the time and effort needed as well.

A “good” & “reliable” Systems Engineering team, that can offer 24/7 support will take around a year to hire and setup, and they need roughly the same amount of time to transition you off AWS in to your system. They probably need closer to 3-5yrs to give the same level of documentation, API’s, tooling, processes, UI’s and training that AWS already provides.

Let’s call it 5 years to get to the level of AWS when you started the transition.

A decent team of 5-7, including engineers + PO/PM + UX and so forth, is at least $1.5M/yr. That’s $7.5M over 5 years, not including your new hardware and networking costs. Let’s call it $10M. You’re also 5 years behind AWS now, and over that transition you’re still paying AWS, and your development speed has halved as you wait for your new team to build or transition infrastructure.

You can trade cost and quality for speed and have everything ready in 2-3 years by setting up a few teams. Add HR support, more contractors etc etc. you’re looking at a $10M+ outlay again, regardless.

Or you can keep paying AWS $5M/yr, renegotiate fees often and literally not worry about that headache and focus on your product.

You act like AWS doesn't require you to have a team, or carefully transition infrastructure. Because of that your cost estimate is much higher than the couple additional people you should actually need.

> They probably need closer to 3-5yrs to give the same level of documentation, API’s, tooling, processes, UI’s and training that AWS already provides.

You don't need to build an internal AWS to manage your own servers.

You are an ISP, and managing that hardware is your core competency. For every other CTO, saving money by moving to on-prem is picking up pennies in front of steamrollers. Just because you spend $3 million/year on AWS doesn't mean that same organization can turn around can hire 10 sysadmins and do everything themselves; anyone who thinks they can simply execute well just because they have a big checkbook is in for a rude awakening.

For a growing company I would always weigh the organizational overhead of moving on-prem to the actual cloud costs. If your goal is in the next year to grow revenue from 50MM to 100MM, then a company wide engineering effort to move to bespoke platform to save 5-10MM just isn't a good use of time (keep in mind you are trying to hire and release new features).

It's not that it's impossible; it's that trying to both scale up your business and provide infrastructure to yourself is just unnecessarily increasing the difficulty on your businesses execution.

I worked at a company that migrated a 100 PB Hadoop cluster to GCP for assorted reasons despite many years of success with colocation. I wasn't involved in any of this, but the team's decision process makes sense. You can read through their decision making in these blog posts:

* https://liveramp.com/developers/blog/google-cloud-platform-g... * https://liveramp.com/developers/blog/migrating-a-big-data-en...

One big point was challenges of maintaining multiple colocation sites, with cross replication, for disaster recovery. Since Hadoop triple replicates all data within one DC, this requires 6 times the disk storage capacity of data size for dual DCs. In contrast, cloud object storage pricing includes replication within a region with very high availability such that storing once in cloud storage may be acceptable. Further, you also need double the compute, with one of the DCs always standing by should the other fail.

HDFS supports RS/XOR erasure coding which gives you same fault tolerance guarantees as 3x at much lower replication factor. This is essentially the same method aws/gcp use under the hood - there’s no magic involved here

Can I use this over NFS or samba or is it just for Java and Hadoop?

Yes you can* for new projects I would go with ceph+nfsv4

*) https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/ha...

ceph+nfs was actually what i was going to do.

Probably some learning pains but at least i don't have to "lose" half my storage because the sizes aren't paired with anything.

If I have to pay the same amount of money and I can choose to deal with people or deal with a decent sized company (and leave them to deal with their people), I always choose the later.

Sorry, that's bullshit.

It all depends on the size of the investment and how you need to run it. I built a "new" environment on a company premises due to some compliance requirements that would be cost-prohibitive in AWS or GCP. The gear was procured through a leasing vehicle, and the hardware vendor had an SLA for delivering compute and storage. HPE happened to win the bid.

There is very little difference operationally. From a costing perspective, it's about 40% less than an AWS solution. But in fairness, the customer had an existing investment in a facility - you'd reduce the savings if you had to lease appropriate space in a colo. There are some differences in terms of headcount, but those staff aren't in NYC/SFO/BOS, so they are very cheap -- senior level engineers for $80-120k, fully loaded.

Startups do stupid shit like buy supermicro computers and cobbling together hardware that gets them into trouble when the mad scientist moves on to a new gig. Makes sense when you're drowning in VC money and need to hire people, but doesn't make sense in most other ways. You avoid that by doing competitive procurements and paying marginally more for HPE/Dell/Lenovo/etc.

> cobbling together hardware that gets them into trouble when the mad scientist moves on to a new gig.

If you think that standards based x86-64 hardware running Linux and ZFS, or FreeBSD and ZFS is something that is super unreliable and requires a "mad scientist", then yes, you are definitely in HPE and Dell's target market.

I laughed at the “mad scientist” comment because every startup I’ve worked for has had a “mad scientist”. They are always very opinionated and have a lot of political capital because of seniority. The weird concoctions they create… the minute they leave all the remaining engineers immediately replace most of it with off the shelf stuff.

Home built web frameworks (which apparently aren’t “bloated” and “slow”), piles of bash scripts because they never heard of Salt (or whatever is the latest config management tool)…

Almost always they think they are “saving money” by doing what they do, rarely do they ever consider the opportunity costs to rolling the entire software stack from the hardware to web stack on their own.

Good times.

There is a survivorship bias at play here. You experienced the startups that survived long enough to need to evolve their tech stack, and presumably the 'mad scientist' tech stack had some part to play in that.

> Home built web frameworks (which apparently aren’t “bloated” and “slow”)

Well, let me show you something: https://code.djangoproject.com/ticket/31624

This is a performance regression of Django's ORM for deleting rows from a table, a query that should be about as simple as it gets:

    delete from tbl;

And yet it's orders of magnitude slower because it somehow generates a bloated query. So yeah, let me tell you, the frameworks that people are using off the shelf–there's a pretty good chance that they're 'bloated' and 'slow'.

> piles of bash scripts because they never heard of Salt (or whatever is the latest config management tool)…

Well, that's kind of the point, isn't it? If you're trying to get a project off the ground as quickly as possible, you kinda don't care about 'whatever is the latest config management tool', you will use what is already in your toolbox that will get the job done.

the op was talking about hardware, I think. but you are spot-on on the mad scientist. we are just in the process to de-clutter all the non standard, hand made, highly customized, never documented, stuff produced buba coworker that, unfortunately, passed away.

btw, we also wrote a HA cluster software for sun solaris in year 2000...

I run 10 years old HP servers, something like DL580G5, with care-packs (extended, paid warranty). We needed to flash firmware, 2 motherboards broke and they sent spare parts to replace. Probably with less enterprise-y server firms it may be difficult to find spare parts after 10 years..

> Colocated systems tend to work very well at first when the original engineers who set it up are all still at the company and it hasn't run long enough to start encountering rare failure modes. They quickly become a nightmare when your engineering staff turns over multiple times and nobody can remember who knows how to do what on the colocated system or if the documentation is up to date or not.

And AWS systems do not have to face the same problem? Is it easier for a new staff to come in and modify a millions worth of AWS system than a colo system?

This all sounds like excuses. Insert that two dog meme, one builds a freaking datacenter using commodity hardware in a barn and the other uses AWS and complains about lock-in and expenses.

As an example, imagine if the founders and engineers of Backblaze thought like that.

I work in a very data heavy HPC space and you are glossing over many things.

* to get performant access to a storage cluster is non-trivial, there are many different variables in place which must be correctly tuned to get good performance. Network topology, high quality NICs and switches, a tuned Linux kernel, client side caching settings, network packet sizes, file system block sizes, erasure coding settings, etc.

* your solution mentions nothing of backups, offsite failovers, or disaster recovery plans.

* your solution mentions nothing of a physical datacenter: fire suppression, battery backups, hvac, power supplies, backup generators, server racks, sound isolation, workspaces for hardware maintenance, network cable routing.

* If you have multiple geolocations you need to have dark fiber or ip transit between locations with multiple ISPs to have high speed connections between sites without downtime.

In addition to the raw costs you have to factor in the lead time of building a qualified infrastructure team, building out the requirements, provisioning hardware and datacenter space, setting everything up, and then tuning everything. With infinite money this is probably still a 2 year lead time at minimum.

I do agree that running multi petabyte workloads in AWS is probably not optimal, but when you are a startup in the growth stage it is probably better worth your time throwing VC money at AWS and building out your product. Eventually, the business should probably migrate to self managed infrastructure once the right product fit has been found and the business is looking to streamline.

You have to factor in the cost of egress from AWS to your nice colocated drive.

welcome to the hotel california...

Last thing I remember, I was

Running for the door

I had to find the passage back

To the place I was before

"Relax, " said the night man,

"We are programmed to receive.

You can check-out any time you like,

But you can never leave! "

And this is why I don’t think AWS will lower egress fees in response to R2. AWS may be more interested in discouraging people from using egress than in capturing the revenue from egress. I predict that, at most, we’ll see a narrowly tailored reduction in egress fees that is designed to be entirely useless for communication between server applications.

We are probably looking at the new future in which Cloud computing == Mainframes of the 50s~80s and fewer and fewer people even know how to run the whole scene on-premise. People who got into cloud computing early (mostly by luck) get to win big bucks and better life styles while others try to dispel the magic from left and right.

ultimately, though, somebody has to own, house and run those mainframes, so it's just abstracting the work away to some other group of people. lots of people made careers out of running mainframes and minicomputers in the 1955-1985 time frame. in the case of things like aws, azure, etc, it's just a lot more centralized in a smaller number of gargantuan companies.

Since the future is pretty much set, I think it's more relevant to try to obtain skills (albeit more difficult to obtain because fewer companies have them) and jump into the wagon.

Engineers who can put together and maintain this hardware don't exist on the job market.

Engineers who can put together and maintain 2PB of storage don't exist on the job market? I'd say you don't know the right people or aren't looking in the right places.

2PB is really not that much stuff these days. It's less than two cabinets of equipment and that amount of space (80RU or so) includes routers, switches, AC power distribution, OOB, etc.

They exist but finding and recruiting those people costs a non trivial amount of time and money. Both of which could be spent on whatever secret sauce your company does.

The thing is, the bar for managing a rack or two of off the shelf hardware is maybe an inch higher than the bar for managing just one server. Is it super hard to find someone that can handle physical servers at all?

Our company is not many magnitudes larger than any company and it is not remotely cost competitive for us to run any of our stack on AWS (heavy data ingest (hundreds of billions of inserts a day,) many disks/'big data' backend, hundreds of customers accessing the data. Even 1 time deals to get us into the door are not cost competitive, let alone long tail economics. Cloud for fast storage and high bandwidth usecases is extremely expensive.

That's a nice statement sounding like straight from Amazon sales reps, and it can actually work for some companies, maybe. But for our bank (top 10 globally), the only way to get even equal financially to our own farms is to have aggressive downtime every night that negatively affect productivity of our global teams. Pricing is really not that great if you deal in scale.

You wanna push 1 evening a bit late to deliver something valuable for the project? Sorry, no can do.

I don't even factor in horribly expensive migration projects that brought actually 0 added business value for the type of apps we use. We still have to keep our Network, Windows and Unix admins, various App support personnel etc., there is plenty of work for them with AWS. Not 1 single IT guy was made redundant.

No cost savings, in contrary.

For a while I did maintain a storage cluster that had close to 0.5 PB of data, and had a capacity of up to 3 PB (if you filled all the slots with the largest disks you can buy). You want to ensure you have a lot of redundancy and spare capacity if you are managing it yourself. Luckily, the hardware is relatively cheap. It's the manpower that costs a lot. Still, I think it was only 0.1 FTE to manage this storage cluster, including the network, file systems, user access, swapping out bad harddisks and storage pods (but granted, it's I/O load was very light). Also, while AWS takes away the burden of handling the physical drives and the filesystem for you, now you have to handle interfacing with AWS. That means you need an engineer that knows how to integrate your application with AWS. If you can leverage more AWS services, maybe even avoid needing your own server rooms because everything is running there, it might pay off. European vs. American salaries might also change the equation. But if you just use it for storage, I don't think it's worth it at any scale.

I'm not saying it never happens, but I've never seen moving to AWS (or Azure, GCP, etc.) save in people costs at any tech company with a large resource footprint. It just shifted where the time is spent and who had to spend it.

The public cloud and managed services work great for the most common use cases, but go outside those and you start having to engineer around limitations.

If you have a sizable footprint in any given dimension you're trading one complexity for another.

The "who had to spend it" is huge - companies love paying providers and hate paying people/depreciating costs.

> hate paying depreciating costs.

One possible solution for this if you want to do it as bare metal you control, is leased equipment (even with $1 buyout at end of term), which can be accounted for differently than purchasing it up front.

Curiously our company moved from AWS to on-premise a couple of years ago. Something about CAPEX -> OPEX was mentioned back then.

That's the second part of the loop, when the cost of AWS is high enough that you can show immediate dollar savings by bringing it in-house.

Yeah, that was a bit more than one year before we filed for IPO :)

It isn’t. I have the same overhead running my server as I do a VM. What I don’t get is a $3000 bill for $100 worth of server.

You can generally buy whatever you are renting from AWS for 1 to 3 months of an AWS bill.

The only thing I don’t get from colo is a bunch of other customers thrashing the cache on my CPUs

Databases are not web servers there’s no possible way to not run a database on smaller / fewer instances when running at non-peak times. Instant scaling is the only possible advantage AWS could bring. However with the prices they charge it’s simpler and cheaper to just buy/rent your own hardware. Especially if you have to pay egress fees. (bandwidth is really the biggest ripoff)

The argument isn’t about using AWS to run a VM (which can be cheaper that coloing your own kit, depending how many you want and for how long), it’s all the extra stuff. Start an aws load balancer rather than run and maintain your own for example.

I don’t like lock-in, but the prevailing view has always been in favour of lock-in, be it IBM mainframes, oracle databases, windows servers etc, and if you swing that way aws has tempting offers.

Oh and databases do scale. Say you want to run end of quarter financials that require a lot of processing for a day, you bring up tons of read replicas and away you go

If you bought hardware with what you pay for AWS RDS you could run your entire DB in RAM. Hell you could probably put the data in memory on a GPU.

Also, this is generally why you run financials overnight. If your hardware is serving transactions during the day it can easily run your quarterlies at night.

nginx is far easier to maintain than AWS load balancers which is what load balancers their load balancers are. The best part about nginx configs? They are cloud agnostic and will work on everything from a Raspberry Pi to a 128 core EPYC server.

I'll tell you something about RDS reliabilty, your monthly maintenance window brings your DB down far more often than a single unreplicated server ever fails. EBS (like the entire thing) has failed more times in the last year on us-east-1 than my colo RAID.

The selling point of AWS is that if you pick AWS and it fails you can say, well the richest guy in the world can't figure this stuff out so it must be impossible, when in reality high school kids could make a more reliable system. If you pick AWS you have the unreliability of the base software / hardware of their systems plus whatever the AWS engineers fuck up. At this point it's pretty clear that they can't even keep a SAN working.

What about all the config management infrastructure to manage those nginx instances?

What is the amount of work required every time some team wants to spin up new stuff? What is the turnaround time between when they file the ticket and the work being completed?

Oh there’s this great thing called files.

You put your config in there and then you can copy them anywhere, even between AWS accounts. They’re even cloud agnostic so when you move to GCP they still work.

Files work great with fucking shell scripts, entirely compatible, it’s rumored that shell scripts beneath the hood are also files. https://github.com/brandonhilkert/fucking_shell_scripts

You can use FSS on docker, kubernetes, bare metal (otherwise known as computers), Windows, Mac, Amiga, VMS etc.

Plan9 which was made by the creators of Go works exceptionally well with files. Some people even say that AWS converts your LB config object to a config file for nginx when it’s provisioned.

If you want to get really crazy you can put all your files in one dir like /etc and then you can use this program grep to search all your configs at once. You can even use things like perl -pi to provide a programmatic interface to editing all your configs at once.

Who writes those scripts? How do you hire and attract talent to manage that stuff? How industry standard is it? Can you train them?

A lot responses here seem to make the embedded assumption that users pay the "advertised price" for storage and compute instances from AWS. When you are at that scale, there are presumably closely negotiated agreements that result in much lower than normal pricing but require large volume commitments and clear ramping projections. Thus the arb between doing it all yourself versus AWS is probably quite different than it might seem from the outside.

Hardware is cheaper at scale too

That argument doesn’t hold up. If how could AWS be cheaper than doing it yourself at that scale? Like if you are a tiny company that can’t afford your own DC, your own engineers, etc. then yes AWS is cheaper in absolute costs but not in per-byte costs. But at scale you should be able to hire engineers and build out a DC at which point you aren’t paying the AWS margin, which is how you save money.

In other words your assessment would only be true if AWS had a 0% or a negative margin.

There is still an economy of scale. You are right, as you use more resources your economy of scale will increase, but it will never match AWS's (unless you are huge). So the math is if the AWS margin is less than the difference between the two different economies of scale then it makes sense to run your own datacenters (ignoring the opportunity cost of the transition costs). Of course at some point the margin will exceed that difference, but depending on what type of infrastructure you need it can be at a very high point.

Your last point is the important part: depending on the type of infrastructure you need you might be able to save money. If you want a cheap place to dump your files, B2 is cheaper than S3 and raw storage hardware pays for itself in about a year. If you need a sophisticated CDN then yeah you’ll need to be huge before it pays for itself. I would consider ditching S3 at the point where I can hire two full time engineers to worry about my storage layer.