News for hackers. Stuff that matters.

A comment on Twitter migrates data to Google Cloud. Parent comment

Re:Twitter migrates data to Google Cloud by parasubvert on Wednesday January 23, 2019 @07:51AM (#18977719)
Re: NUMA, I had thought it was fairly straightforward to peg VMs to particular NUMA nodes (or declare CPU or memory affinity) in VMware. I agree GCP does it well (better than the other public clouds!), but I've also seen this done well on-prem.
[ Parent | on HN ]
Re:Twitter migrates data to Google Cloud by dijit on Wednesday January 23, 2019 @08:02AM (#18977756)
I have too, which is why I singled out hyper-converged.

In hyper converged environments the storage and the compute live together on a service mesh and thus some processing is done on the hypervisor to manage storage, those processes don't have an affinity (in the case of vmware for example) so a guest VM core gets suspended for a couple of hundred clock ticks sometimes.

[ Parent | on HN ]
Re:Twitter migrates data to Google Cloud by _msw_ on Wednesday January 23, 2019 @12:32PM (#18980358)
Hi! I'm from the EC2 engineering team.

This is a super challenging problem for general purpose virtualization stacks. EC2 has been working to avoid any guest VM interruptions. With the Nitro hypervisor, there are no management tasks that share CPUs with guest workloads.

See more here, including data from a real customer with real-time requirements running on EC2 under the Nitro hypervisor: https://youtu.be/e8DVmwj3OEs?t=1796

[ Parent | on HN ]
Re:Twitter migrates data to Google Cloud by parasubvert on Thursday January 24, 2019 @09:38AM (#18989002)
That's a good point, I have seen cases where we've separated a portion of nodes for VSAN or ScaleIO out to a separate cluster to avoid this behaviour on latency sensitive workloads. These nodes don't need a ton of RAM and you'd better have good E-W bandwidth for them, but it's always a tradeoff...
[ Parent | on HN ]
Re:Twitter migrates data to Google Cloud by _msw_ on Wednesday January 23, 2019 @12:23PM (#18980272)
Hi! I'm from the EC2 team.

This is a great tool to help uncover if CPU and memory NUMA within a VM aligns to the physical host or not:

https://www.cl.cam.ac.uk/research/srg/netos/projects/ipc-ben...

Paper from the USENIX 2012 conference: http://anil.recoil.org/papers/drafts/2012-usenix-ipc-draft1....

EC2 instances have been NUMA optimized since we launched our CC1 instances in 2010. I encourage you to try ipc-bench on your cloud provider of choice, if CPU and NUMA affinity is important to your workload.

[ Parent | on HN ]