Everybody asks why we don't do cloud, and time and time again i have to point out that the fastest per-thread cpu on ec2 costs more per month then our rented bare metal server, that benchmarks as faster per thread, and has 3x the bandwidth we need, included in the price!.
Even on a cpu level, the obsession with intel """enterprise""" cpus is leaving money on the table.
"but what about web stuff, surely that could go on the cloud!" the best cpus to run (mostly) single threaded game servers right now come with 8 to 16 cores, and we aren't running that many game servers in each region we target, so what else are we gonna do with those spare cores but load up ansible managed docker swarms to handle the webstuff?
Notice how I haven't even needed to get to the egress cost to point out how senseless going to the cloud can be.
For example, Swarm secrets can only be exposed as files, not as environment variables. You can argue this is more secure because file permissions are more granular than env vars, but IMO that's a silly argument in a container context because containers are almost always single-user to begin with. Moreover, the vast majority of containerized applications expect their secrets in env vars, so you have to resort to fragile entry point wrappers if you want to make use of Swarm secrets.
automating secret usage across containers that are deployed via ansible scripts gets annoying in that situation.
If there is a fourth stage then it’s getting to the point that it’s more profitable to run a side business renting out your unused resources to others and you become a cloud.
I’m surprised there isn’t a Walmart cloud or an Exxon cloud by now.
This is a common belief about AWS, but I find the counterpoint persuasive - if AWS involved Amazon renting out its unused resources, it would have to shut down in December. But obviously AWS does not shut down in December. They're not renting out their unused resources; they're renting out their experience running this kind of system.
1. On prem hosting managed by 1 person who works out of broom closet in some budget colo. Zero compliance or official procedures involved.
2. Cloud hosting managed by whoever needs a thing taken care of. (We are here)
3. On prem managed by dedicated in-house team in multi-colo setup. (We really really want to be here)
I think we will skip the 4th stage. If we are forced to build our own datacenters, something went very badly.
The 30k/m figure posted elsewhere in this thread is a fairly reasonable line in the sand for jumping from 2 to 3. We also have unique regulatory requirements that could move that threshold lower.
another major advantage of this kind of setup is that it is possible to give far better (technical support) to your customers, especially if you provide infrastructure services to your customers.
having a competent technical team which can do proper indepth analysis of issues customers are having has a massive positive impact on customer relations. Compared to most technical escalations i have seen at most of the cloud providers, which comes down too "it was an internal problem, but is is fixed now, don't know if this will happen again".
This is exactly why we want to do it. Being able to offer our customers a 100% vertical solution stack that we completely control & understand is a massive selling point in our industry (and hopefully others).
at $job, me and another team member basically did technical deep dives into issues that occured for $customer. Usually they needed deep root cause analysis as of why something happened. (Mainly for insurance/legal/compensation reasons).
It was a ton of fun and we even got to a point where customers where willing to pay for this service when required. It costs us a lot of hours to do the troubleshooting, and not all customers have such requirements.
Some fun root causes i have found over the year:
- a bug in a firewall which resulted in kernel crash of the packet processing plane (and thus packet loss). We worked extensively with the vendor to fix this. This bug only occured because $company was using a custom TCP stack, and was sending ACK's incorrectly every X packets.
- tons of issues regarding time. Time drift on routers, time drift between processes inside a router. Time drift between hosts.(this happens a lot, and it makes me appriacate the fact that some routers have external clocks as a source option).
- quite a lof of issues with IPsec tunnels. For being a standard, i have never seen suchs a lot of edge cases with non-interoperability.
We pay a 3rd party for our AWS (and Azure) usage to get better rates than AWS would give us (and without a commitment to increased spend like AWS reps have tried to push for Amazon's available programs).
I wouldn't be surprised if they just used volume discounts / negotiated pricing and reservations at their end to make it massively profitable for them.
And if you are present with your own bare metal infrastructure at such a place you almost certainly also have the opportunity to connect to a serious IX for settlement-free open peering, and to run PNIs to other major sources or sinks of your traffic. So by no means will all of your traffic be going through transit.
At that point you’re probably buying dark fiber to get where you need to go.
Even with discounts from AWS it just doesn't make sense to use AWS to serve up the assets to users.
We do use S3 as our static asset backend and the combination works really well. I would love to see Cloudflare release a S3 compatible storage service though. I think we'd jump onto that in a heartbeat.
"Lifecycle rules instruct the B2 service to automatically hide and/or delete old files. You can set up rules to do things like delete old versions of files 30 days after a newer version was uploaded."
CloudFlare's CDN is competitive on features and price but it isn't a market leader in terms of raw speed.
I spent a big chunk of my last 2 jobs dealing with that issue.
even in europe, AWS has "meh" regions compared to azure or many of the colo/dc locations.
Doesn't mean they couldn't do some optimization! But to your point, yeah, I imagine there's probably a little more pressure to develop new features/characters/etc. rather than spend months rigging up in-house network infrastructure.
I can see how running game servers—with a (somewhat, maybe?) predictable load of players—can be done efficiently on baremetal and with colocating.
For another business that has huge peaks of demand, such as analytics with dynamic queries, I fail to see how baremetal can compare to spinning up hundreds of instances on-demand.
Perhaps it comes down to what services you offer your clients and how you implement them?
If the cloud to bare metal price difference is 10x.. you could easily just buy bare metal = 2x your peak load, and still come out ahead..
And that's why I come to HN, lobsters, etc :) so that I can read and learn from others' experiences...
Google really does need a service that can scale to 100M users.
I doubt someone starting a SaaS to make garbage truck software needs to. Or at a minimum, at the time they need that kind of scale, they will have plenty of money to throw at hardware. It's amazing how many users a single maxed DB can handle.
If you're trying to reduce costs on a complicated usage pattern, then you'll probably want to have someone on staff or consultant who knows how to deal with those things.
I don't know your experience, but bare metal has at least an order of magnitude greater efficiency at one or two magnitudes lower cost. If you want to pay $1MM/mo for something that could also be done for (generously) $20K/mo (plus a couplefew salaries that add up to much less than $1MM/mo), that's your business, but I don't see how the use-case you describe requires cloud/AWS.
The thing that people don't get (or have forgotten, or...) is that you don't have to spin up hundreds of instances in the way you do on AWS, you just handle them on a machine or three that already has enough headroom to handle them.
(all hypothetical blather shaped to support my perspective and disprove the utility of yours, of course ;)
Not so sure about most people learning this...though, I will say I'm probably a late bloomer compared to most on HN :)
What you state makes sense. I think my tone was dismissive of the idea that running your own machines is worthwhile. To add to that, I haven't (yet) worked on large-scale systems so my comment is purely speculation.
If you look at what it would cost to run bare metal with top shelf companies, it's often as cheap or cheaper than spot/preemptible instances at the cloud shops.
It was a system that had fixed scale, high licencing costs and did not bridge well into software that increasingly leveraging the AWS environment. Egress was least of our worries relative to integration complexity in services ecosystem.
You have to be a very very large scale before self managed services make sense; as you suffer at integration layers for every developer that is even an inch off the well beaten path.
95th billing is the way to go. Due to the nature of the company's content (adult) nobody wanted to do peering agreements due to the public outcry if anyone found out but if that had been possible I figure the plan would have been to follow Netflix's open connect journey.
How to run a top 20th largest bandwidth consumer in the world for < $1M/month. https://openconnect.netflix.com/en/
Disclaimer: These are only my own personal opinions
You are not AWS's target market.
In that case, the economics of bare metal aren't as clear-cut.
I run a team that manages our public and private (DC) cloud for a successful tech company. - I need more folks to help us do things better. Please contact me Mr. Pugs or others if you are open to thinking about looking at another company.
People are swimming in the koolaid and don't think a different environment exist.
These kids. When I was a kid......
With AWS I could just spin up 10 Kubernetes clusters with pretty much unlimited resources, can't do that in OpenShift because you'd hit a resource quota or limit.
DigitalOcean Egress - $0.01/GB
why?
Last I checked, AWS wanted ~$200k per MONTH, for just bandwidth (no compute, memory, storage, or anything else). We'd love to be able to use the cloud, but not at the cost of increasing our monthly expenses by an entire order of magnitude, so we just stick to bare metal colocation.
I honestly believe that if you have the technical skill in-house, and are spending more than, say, $30k/mo on public cloud hosting, that you should seriously evaluate whether bare metal could significantly decrease your costs.