Also surprised about latency, latency from what to what? Big cloud providers have excellent globally spanning networks. Long distance networking is crazy expensive, though, compared to the peanuts it costs to transfer data within a data center.
Reliability - again, not sure I buy it. Reliability is "solved" at low levels (such as data storage), most failures occur directly at service levels, regardless of whether you have the service in house or in the cloud.
The rest of your points make sense.
How much would it cost to move this across boxes in EC2? I actually don't know, that's not a rhetorical question. A lot of our servers have 10-40gbit links that we saturate for minutes/hours at a time, which I suspect would be expensive without the kind of topology optimization we do in our datacenters.
> Also surprised about latency
We've spent a surprising amount of money reducing latency :) We're not a high frequency trading firm or anything, but an extra 1ms (say) between datacenters is generally bad for us and measurably reduces performance of some systems.
> Reliability is "solved" at low levels
To whatever extent this may be true, it's certainly not true for cloud providers. One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box. This would cost us a lot of money (mostly in dev time, to work around it).
Also, multi-second network dropouts in big cloud datacenters are not uncommon (in my limited experience), but that would be really bad for us. We have millisecond-scale failover with 2x or 3x redundancy on important systems.
Data transfer between instances in the same AZ is free. If the data crosses AZs, you're changed $0.01 per GB in both directions. This is for instances on a VPC. I think the pricing model is different for classic EC2.
There are some exceptions like all traffic between EC2 and ALBs being free.
Edit: Pricing is described at https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer
I think everyone does something of this sort nowadays, that's why networking is ~free within data centers :)
> but an extra 1ms (say) between datacenters is generally bad for us and measurably reduces performance of some systems
That speaks to me. You will always be just the n-th client unless you own the cross-datacenter data links (i.e. have full autonomy on deciding the priority of the traffic). It's similar to the covid provisioning problems you had mentioned.
> One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box.
Yeah like others pointed out - that's just what "cloud" is, and is generally a good idea. You're supposed to handle a certain % of your machines going dark without a warning without violating any SLO (or even worse, certain % of your machines "pretending" they're up but actually being ridiculously slow for this or that reason; and don't even get me started on CPU/RAM bitflips).
It sounds to me that you run an extremely highly sensitive service, something for which paying for true ownership of the hardware just makes sense to remove those kinds of risks that most services don't care about. At the end of the day "cloud" is a shared resource, and no resource separation efforts will be 100% effective.
Nothing. You generally only pay for data going out of cloud providers. Not data going in or data being transferred within the same region.
> One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box. This would cost us a lot of money (mostly in dev time, to work around it).
You're not going to have a successful cloud experience unless you build your applications in a cloud suitable way. This means not all legacy applications are a good fit for the public cloud. Most companies really embracing the cloud are mitigating those risks by distributing workloads across multiple instances so you don't care if any one needs to be restarted, especially within a planned window.
> Also, multi-second network dropouts in big cloud datacenters are not uncommon (in my limited experience), but that would be really bad for us. We have millisecond-scale failover with 2x or 3x redundancy on important systems.
Are these inter-region network dropouts or between the internet and the cloud data center? You're not going to be relying on a public internet connection to the cloud for critical workloads.
All that being said, there are plenty of workloads which I don't think fit well in the cloud operating model. You may very well have one of them.
Right, we have not done this. We basically decided it was cheaper to keep doing the “old school” thing and not spend a bunch of dev time trying to do it in a way that supports arbitrary failure of N boxes. We just spent the money to make it unlikely our boxes or networks will fail, and if they do fail we may be sad (but it rarely happens, and has not yet happened in a catastrophic way).
> Are these inter-region network dropouts or between the internet and the cloud data center?
I’ve seen multi-second dropouts within a DC (cloud, not my current company), and multi-hour single-path failures between DCs (usually something like a fire or construction cutting a line). But all of our DCs have at least 2 physically independent routes to the internet, so it’s never taken us fully offline.
Our fault, I suppose -- but multi-az is prohibitively expensive if you need to run anything data heavy distributed.
> but multi-az is prohibitively expensive if you need to run anything data heavy distributed.
If you communicate between your AZs via ALBs, multi-az is effectively free. Our bill is so high because within our Kubernetes cluster, our mesh isn't locality aware; it randomly routes to any available pod. 2/3rds of our traffic crosses AZs.
It's nobody's fault that the billing structure at Amazon is so complicated and confusing.
Except Amazon's.
This is not true, AWS charges for all traffic cross AZ, so if you want a resilience within a region (which you mentioned in next paragraph) you will do a lot of that talking and this ends up a non trivial cost. You can do some optimization though by having your apps being aware of AZs they are on, but most places don't do that, more common are places that run everything from a single AZ.
It's interesting that you say this, because I think this just boils down to how you treat failures. Boxes will fail. It's inevitable, even if you run your own hardware.
We treat failure as an every-day event and have set up our systems so it doesn't matter. One box fails, an automated system notices and brings up another to replace it, while the remaining boxes take slightly more load for a few minutes.
Sure, that kind of failure tolerance doesn't come for free. But, again, you're going to have failures, no matter how much work you put into reliability (which also doesn't come for free!).
I believe I saw a slide that average lifespan of an EC2 instance at Netflix is 14 minutes.
I’m not necessarily saying cloud will work for your setup, but you can’t compare like that.
What specs are those? I was under the impression AWS has everything from extremely tiny to giant terabytes-of-ram-for-SAP instance types.
Amazon is constantly expanding the selection, so it's possible they've added something since we last seriously looked into this.
For most every workload I can think of, as their load increases it's always one resource in particular that's the limiting factor. (RAM/CPU/Storage/etc). So it makes sense to me that AWS instances focus on optimizing one particular resource type.
Would be interesting to hear about types of workloads that break this pattern. (Or maybe it's a few different types of workloads/services that need to be tightly integrated on one machine?)
In my experience, harder to also balance I/O so you’re close to hitting three limits, but update-heavy transactional workloads can manage it for disk-based DBMS’s.
So this is a rediscovery, at least an example, of what could be called the bottleneck principle of system performance analysis and optimization! Sooo, just look for the bottleneck(s), work on those, and f'get about everything else until, say, everything is equally a bottleneck at which time have a well balanced, call it optimized, which can be appropriate, configuration!
At times, e.g., at IBM's Watson lab, there has been a lot of work on applying queueing theory, analysis, and simulation to analyzing and then optimizing such systems, but, more closely to fully true than one might guess, all that really mattered were the bottlenecks(s)!
What industry are you in?
Are you concentrated in 1 geo location or... across USA / across 1 county / global?
Something plumbing-y; the company is not well known
> 1 geo location or...
Global, although probably 50% in the US
Haha. This can't possibly be true.
I've seen on-premises, where reliability wasn't even objective machines that easily run for 5+ or more years without any interruption. Now parent said that they actually need reliability and you can achieve it using many technologies. Starting with RAID, dual PSU, or even hot swappable RAM or CPU (I remember SPARC machines allowed this). With full control of networking you can also make a standby node take over nearly instantaneous, when in AWS it might take couple minutes. You can achieve nearly any availability as long as you have enough money. In AWS you don't have any control and your only way is through designing your application in specific ways and that still has limitation. Just take look at RDS when everyone would want to have it instantaneous, but it usually will take few minutes.
Consider that building fault-tolerance into their application may be very difficult, or impossible. Cloud would be incompatible with that.
We don't generally trust cloud providers to meet our requirements for:
* uptime (network and machine - both because we are good at reliability [and we're willing to spend extra on it] and because we have lots of fancy redundant infrastructure that we can't rely on from cloud companies)
* latency (this is a big one)
* security, to some degree
* if something crazy is happening, that's when we need hardware, and that's when hardware is hard to get. Consider how Azure was running out of space during the last few months. It would have cost us an insane amount of money if we couldn't grow our data centers during Corona! We probably have at least 20-30% free hot capacity in our datacenters, so we can grow quickly.
We also have a number of machines with specs that would be hard to get e.g. on AWS.
We have some machines on external cloud services, but probably less than 1% of our deployed boxes.
We move a lot of bandwidth internally (tens of terabytes a day at least, hundreds some days), and I'm not sure we could do that cheaply on AWS (maybe you could).
We do use <insert big cloud provider> for backup, but that's the only thing we've thought it was economical to really use them for.