The difference between hypervisors for IO is tiny. They're all slower than native, but vary very little between one another. That's why the manufacturers have all almost given up trying to use performance for comparison, and instead argue value adds/management/automation/cost.
If you really found one specific hypervisor slower than another (regardless of which two), I'd want to look at your exact test conditions since that has never been the case for me (everything else being equal). And even the manufacturers don't argue that point much.
There are plenty of reasons to e.g. pick AWS over Azure, but Hyper-V performance is not one of them. That's just FUD.
For example, we found two orders of magnitude performance difference between PV and HVM on ec2 on fork-heavy CI jobs. A difference between 3 and 30 minutes. This difference is trivially reproducible by measuring the time taken to complete a fork() syscall. As the memory size of the parent process grows, the difference becomes more pronounced.
Here's actual data I collected from c3.large instances, using a contrived Ruby one-liner to benchmark fork():
PV: [ec2-user@PV ~]$ ruby -e'$ref=[]; (24..29).each {|n| $ref << "X" * (2 n); start = Time.now.to_f; fork.nil? and exit!; puts Time.now.to_f - start; sleep 1}'
0.00550532341003418
0.011992692947387695
0.02542281150817871
0.05213499069213867
0.10562920570373535
0.21283364295959473
HVM: [ec2-user@HVM ~]$ ruby -e'$ref=[]; (24..29).each {|n| $ref << "X" * (2 n); start = Time.now.to_f; fork.nil? and exit!; puts Time.now.to_f - start; sleep 1}'
0.0005323886871337891
0.0007219314575195312
0.0012335777282714844
0.002183198928833008
0.004210233688354492
0.008102178573608398
As you can see, the difference is considerable. This is all on Amazon ec2, the only difference between these VMs is PV versus HVM.
(edit: HN formatting seems to be eating the exponent operator in the "2 to the n" expression. It's a double *)
also note that fork() involvs "lots of" io computations which could also be a driver problem and they evolved in newer kernels, too. and also fork() could syntactially be different on hosts with ballooning and hosts without. especially on kvm.
edit: as of today we switched to hvm instances (however we are on many many micros so that performance boost is likewise zero) on our inhouse hardware we are varying between kvm and vmware. however I dislike vmware more and more. vcenter is just unusable (flash) and consumes a shitload of memory. so mostly we use vmware api calls which makes vmware unnecessary somewhat. also we sell our software as a appliance with a xenserver host and mostly we tested our software an all ends and we are mostly disk io driven and that barely changes between them. there are some aspects that could improve performance but that depends more on how you configured your virtual disk.
32bit, or 64bit pre-virtualization hardware extension (vt-x/d, ept, amd equivalents) are really the only ways this should have been the case. Each time you make a fork() system call on 64bit PV you're making an extra context switch and flushing the TLB that an HVM instance does not need
I would expect a full description of the benchmarks and setup to follow up.
After reading your comment, and admittedly not knowing much about hypervisors (yet), I am genuinely curious as to what else could be causing this drastic difference in performance, if not for the hypervisor itself. Any ideas?
Real hypervisors are utilising a lot of hardware features to accomplish jailing. Client hypervisors may or may not be, depending on how they're implemented, and or how they're integrated into the underlying OS (e.g. specialised drivers, just simulating hardware features, etc).
You should be looking at dedicated virtualization hosts to do an apples to apples comparison, like vmware on bare server metal vs hyperv or any of the free tools also on metal. Not on a cpu throttled desktop.
I would always separate benchmarks for server or desktops.
Network on Azure though is a totally wrecked thing:
- A0 instance is capped to a miserable 5 megabit/s uplink (downlink is fine);
- If you want TCP/UDP ports open, you need to open them one-by-one, going through a 2-page wizard with 8 fields in total to fill in for each port you are opening, and then waiting 15-20 seconds for each "change" to be applied. Good luck opening a 1024-port range for your SIP server;
- You can't open ICMP/GRE/IPSEC or any other custom protocol;
- There are ton of really weirdly configured network gear between your VM and the Internet - packet drops and private IPs in traceroute, etc...
- On a plus side, you can enable so called "direct server return" and you will get incoming packets with your public IP as a destination IP; a really nice feature sorely missing on AWS.
I thought it was 10Mb/s?
>If you want TCP/UDP ports open, you need to open them one-by-one, going through a 2-page wizard with 8 fields in total to fill in for each port you are opening, and then waiting 15-20 seconds for each "change" to be applied. Good luck opening a 1024-port range for your SIP server;
...or use the powershell cmdlets and do it in one line
Alas... Also, in my experience, throttling is really slow to react, i.e. you get a chunk of data out at 10-20Mb/s, then everything freezes for few seconds until average would drop below 5Mbps, then another speed-up/slow-down cycle. Long-term average is exactly 5Mbps as per their spec-sheet [1].
[1] http://download.microsoft.com/download/4/1/1/411621F0-D0BF-4...
[2] https://www.oaklight.us/2014/06/azure-network-speed-quick-te...
Many large providers are numbering their hops with private IP space and routing public space across it because they don't want to waste /30's of public space for each point to point.
Yes, /31 is an option, but there were a lot of edge cases or network gear that didn't support it.
Which bring a related question - what's up with none of AWS/Azure/GCE supporting IPv6?
There are some things wrong on your comment, please let me comment on them:
I don't think A0 are capped at all, I don't know where did you get that:
user@A0VM:~$ speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Microsoft Corporation (40.113.XX.XX)...
Selecting best server based on latency...
Hosted by KsFiberNet (Wichita, KS) [45.26 km]: 139.703 ms
Testing download speed........................................
Download: 44.48 Mbit/s
Testing upload speed..................................................
Upload: 22.54 Mbit/s
I just had to run one single test to get these results. Multi-threading will probably pump this up (let's say a multi-thread iPerf test between two VMs on two different regions, through their public IP addresses).
(Funnily enough this VM is in Dublin, hence the latency to Wichita).
You can get access to all your TCP/UDP ports by adding an instance-level IP: https://azure.microsoft.com/en-gb/documentation/articles/vir... (you can later protect it with NSGs --> https://azure.microsoft.com/en-us/documentation/articles/vir... ). What you've seen is when you create endpoints on the load balancer that points to the VMs behind it. Now in Azure Resource Manager, spinning up a VM from the new portal will actually get you an ILPIP and no load balancer by default.
About that statement about a ton of weirdly configured network gear: Well, for starters you're doing network virtualization here :) but seriously now, the platform doesn't forward ICMPs to/from Internet, so I'm curious about those private IP addresses on a traceroute (and of course the packet drops). If you're talking about hops between you and the VIP (the load balanced public IP in front of your VM), that's probably your provider, other providers and finally Microsoft's Network. As per the rest of the path, you've said that ICMP doesn't go through, so traceroute it's not going to work (tcptraceroute also needs ICMP to work :)).
Speed test though... hm... 22Mbit/s is not exactly unrestricted, plus not sure how long is the burst that "speedtest-cli" makes. In my experience A0's uplink is burstable (but unusably slow when recovering from a burst), with long-term average being bang-on 5Mbps.
I do of course see exactly one "my" private IP in traceroute, but I'm not talking about that. I meant there are microsoft's private IPs (#12-#15) and dropped packets (#16 and on).
And of course, "platform doesn't forward ICMPs" is an annoyance in itself - i can't even ping my own VM.
# target IP anonymized, but still have enough granularity to reproduce the traceroute
traceroute to xxxxx.cloudapp.net (191.235.128.128), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) 0.093 ms 0.117 ms 0.063 ms
2 193.47.232.12 (193.47.232.12) 0.785 ms 0.785 ms 0.776 ms
3 microsoft.mix-it.net (217.29.66.112) 1.219 ms 1.233 ms 1.225 ms
4 xe-0-1-1-0.fra-96cbe-1a.ntwk.msn.net (207.46.42.12) 8.700 ms 8.902 ms 8.430 ms
5 * * *
6 * * *
7 104.44.9.142 (104.44.9.142) 42.112 ms 39.194 ms 38.987 ms
8 * * *
9 be-6-0.ibr01.dub30.network.microsoft.com (104.44.4.142) 39.738 ms 38.266 ms 38.929 ms
10 * * *
11 ae2-0.db3-96c-3b.ntwk.msn.net (204.152.141.81) 36.986 ms 36.984 ms 36.925 ms
12 25.149.64.247 (25.149.64.247) 36.951 ms 36.775 ms 37.152 ms
13 10.10.132.81 (10.10.132.81) 36.917 ms 37.477 ms 37.287 ms
14 10.60.31.77 (10.60.31.77) 37.317 ms 37.699 ms 37.899 ms
15 10.60.31.69 (10.60.31.69) 37.653 ms 37.326 ms 37.441 ms
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
P.S. #12 - lol!In view of today's news, this is not funny at all.
netname: UK-MOD-19850128
descr: UK Ministry of Defence
country: GB
org: ORG-DMoD1-RIPE
admin-c: MN1891-RIPE
tech-c: MN1891-RIPE
status: LEGACY
mnt-by: UK-MOD-MNT
mnt-domains: UK-MOD-MNT
mnt-routes: UK-MOD-MNT
mnt-by: RIPE-NCC-LEGACY-MNT
created: 2005-08-23T10:27:23Z
last-modified: 2015-07-24T14:31:16Z
source: RIPE # FilteredI generally work with Azure CLI (https://github.com/azure/azure-xplat-cli) and ARM templates (https://github.com/Azure/azure-quickstart-templates), helps with simpler configuration and consistent results across deployments.
Though we are aggressively working on adding more features and improving documentation, evidently there are gaps in certain areas. Please feel free to reach out to us through Twitter (@AzureSupport) or Stackoverflow (tag-Azure), for anything we can help with your deployments on Azure.
My comment about the traceroute was because you previously said that you've found those weird IP addresses between the VM and Internet. That's not how I actually see it, but I get your point, too.
The ones you identify as packet loss are just the ones you didn't get an ICMP back. I don't know anything about your background, but as per my experience with networking I've stopped considering traceroute as something remotely useful further than the first hop long ago, especially on environments outside my control (i.e. Internet). There are things out there like ICMP throttling and the fact that I could send you the "TTL Expired" ICMP identifying myself with any IP address. Remember the "Star Wars traceroute"? http://www.theregister.co.uk/2013/02/15/star_wars_traceroute... (doesn't seem to work for me now).
(About #12, WTF? Weird!)
> (About #12, WTF? Weird!)
There is a fairly boring explanation to that. Similar to public IPv4 exhaustion, RFC1918-style private IPv4 addresses can easily be exhausted in a big private network as well. If you have noticed, MS has actually run out of RFC1918, and uses RFC6598 (a.k.a. 100.64.0.0/10) network in Azure for their "classic" segment. 100.64.0.0/10 was not supposed to be used as RFC1918-style general-purpose private IP space, but rather as a local side of a CGN pool. But then again, if you squint a bit, well... it's still private, so will never be advertised into the Internet, so it's fair game to use it if you run out of RFC1918, right?
Some people choose to squint even further, and declare anything that is not advertised into public BGP as a fair game. I've seen a big ISP "borrowing" 44.0.0.0/8 (amateur radio AX.25) subnet to deploy video segment of their huge "triple-play" network. 33/8 (US DoD) is also popular as "borrowed" private space, 1/8 was fairly popular until it actually became real public address space handed to APNIC (oops!). Now we have an example of using 25/8 like that as well.
Currently working at a large MSP, and all of our stuff is still IPv4 only. Although we happen to have a ton of extra IPv4 space, so we aren't as concerned, but there is no movement towards IPv6 at all.
Never used the cmdlets for Azure and it's been a while that I dealt with it, but that sounds like an issue you'd only do via the web interface if you're insane.
Thanks for pointing those out!
The game are tools like Kubernetes and Apache Mesos. I have X thousand VMs spread out through the world, with a few hundred per datacenter for geo-stability.
If a network link fails, no biggie. Still runs. If a rack dies, still no biggie. Redundancy makes it work. Earthquake/tsunami/natural disaster ? Your stuff still runs, with a bit higher latency and slower throughput.
There are a great number of variables that come into play with configuring each hypervisor. Also, considering continual updates, a solid analysis 6 months ago may be stale. Not to mention most of the studies come from the vendors, not from independent parties.
Question: does anyone know of an ongoing/reliable performance review of the major hypervisors?
I'm not sure of any way to get a console through IE. Can you explain?
(1) https://technet.microsoft.com/en-us/library/dn282274.aspx
The basic I/O, compile & runtime performance was significantly inferior to xen & kvm (didn't bench against vmware, phasing it out due to cost), it wasn't worth any effort to even deploy apps for testing.
Therefore I don't see Linux on hyper-v being a compelling option for the cost conscientious technical officer or lead engineer.