The fixed performance EC2 instances (in contrast to instances with burstable performance) all have dedicated physical CPU allocations, and 1:1 affinity from vCPU to the underlying CPU. Similarly, memory allocations are fixed, pre-allocated, and are aligned to the NUMA properties of the host system. When combined with both our Xen hypervisor and Nitro hypervisor, this provides practically all of the performance of the host system with regard to memory latency and bandwidth.
See: https://twitter.com/_msw_/status/1045032259189760000
With our latest 25 Gbps and 100 Gbps capable instances, EC2 to S3 traffic can make use of all of the bandwith provided to the instance.
Please send me more information about the interaction you had on the EBS side, msw at amazon dot com. All writes to EBS are durability recorded to nonvolatile storage before the write is acknowledged to the operating system running in the EC2 instance. fsync() does not necessarily force unit access, and explicit FUA / flush / barriers are not required for data durability (unlike some storage devices or cloud storage systems). Perhaps there was confusion about the question that was asked.
In hyper converged environments the storage and the compute live together on a service mesh and thus some processing is done on the hypervisor to manage storage, those processes don't have an affinity (in the case of vmware for example) so a guest VM core gets suspended for a couple of hundred clock ticks sometimes.
This is a super challenging problem for general purpose virtualization stacks. EC2 has been working to avoid any guest VM interruptions. With the Nitro hypervisor, there are no management tasks that share CPUs with guest workloads.
See more here, including data from a real customer with real-time requirements running on EC2 under the Nitro hypervisor: https://youtu.be/e8DVmwj3OEs?t=1796
This is a great tool to help uncover if CPU and memory NUMA within a VM aligns to the physical host or not:
https://www.cl.cam.ac.uk/research/srg/netos/projects/ipc-ben...
Paper from the USENIX 2012 conference: http://anil.recoil.org/papers/drafts/2012-usenix-ipc-draft1....
EC2 instances have been NUMA optimized since we launched our CC1 instances in 2010. I encourage you to try ipc-bench on your cloud provider of choice, if CPU and NUMA affinity is important to your workload.
However it may depend on instance type and other factors. This complexity is a problem with AWS.
Live migrations is amazing.
Now if only Google paid more attention to certain features of their GLB... amazing tech with showstopper bugs means amazing tech I can’t use. (Also means it’s not so amazing)
"We have important news about your account (AWS Account ID: XXXX). EC2 has detected degradation of the underlying hardware hosting your Amazon EC2 instance (instance-ID: i-XXXX) in the us-east-1 region. Due to this degradation, your instance could already be unreachable. After 2018-12-28 19:00 UTC your instance, which has an EBS volume as the root device, will be stopped."
The stop date is 2 weeks after the email is sent, but as the message states, the instance is likely already dead. It was already dead in this case. GCP will live migrate your instance to another host.
Like the parent said; we used to receive these notices quite frequently and the amount of time varies wildly. I should ask the teams who currently/primarily use AWS perhaps the situation has changed?
But it was not uncommon to have 24hrs of notice, especially for instances in US-EAST2 for some reason.
https://aws.amazon.com/blogs/aws/the-floodgates-are-open-inc...
So, obviously my experience is based on my use-case so things that are important to me are predictability of resources and consistency of data.
From a resource perspective (GCP:Compute) there seems to be a tendency of setting CPU affinity for certain cores, often in the same NUMA zone on the host. This means my tickrate doesn't get hijacked on certain cores randomly. That's quite nice honestly and it's something that on-prem hyperconverged VM solutions don't manage. The NUMA zone affinity / samezone affects our performance quite drastically as certain applications (such as in-memory databases or gameworlds) allocate large chunks of memory and then do little updates millions of times a second. This means memory bandwidth is very important.
GCP also has live migration of instances which is 100% transparent to the application (even the CPU clock seems to be moved over), unlike my colleagues on AWS who get 24h notifications of host machine maintenances I've never had to "deal" with even a single instance outage since starting up on GCP 1y ago. But I would assume if AWS hasn't moved on solving this problem they will soon.
Things like VPC networking being inter-zone by default is something that just makes sense and solves some headaches that I hear other teams having.
From a storage performance perspective, on AWS we were hard-capped to 5GBps per instance to S3, such a hard limit does not exist for GCP instances to cloud storage so we're able to make use of it. (in fact, I can often saturate my 10GBit interface limit even to non-google instances). A n AWS sales rep that this hard-limit can't be removed no matter how much we pay.
I can't attest to the difference in APIs, since I use terraform and that abstracts away anything I would be using.
Regarding storage again; we had reps from AWS talking to us at length and they would not guarantee that fsync() would be honoured on elastic storage. They stated that "it should never be needed" but honestly I'm not comfortable with that answer. I mean, my bare-metal database instances haven't gone down in 3 years but it doesn't mean I'm going to start decreasing the consistency for raw throughput.
There's a lot more but these are the most important things that I can think of right now.