Worse still, we spent weeks on the phone with AWS insisting there was no throttling - just to find out there was throttling. And that would have been discoverable if a single engineer had looked at CloudWatch.
Do you have Enterprise Support? If so, this is a very surprising result.
Just Business Support, we're not that large of a shop.
> They also have hidden limits on the number of incoming tcp connections you can have.
Do you have proof of this?
No, I had an experiment I was tried to run on EC2 in 2013, and ran into this. It was very clear though. Established connections would plateau, and then no more tcp syns would arrive to the ec2 host, unless a connection was closed. At the time the limit was 5000 connections on the micro instance -- from my notes, we got an allocation for hi1.4xlarge and I know we hit the limit there too, but I don't recall what the limit was. This was a very simple TCP proxy, using Linux kernel ipmasq; this uses significantly less memory than HAProxy, although with a lot less features.
We had a very excited account rep because of where I work, but he was only barely able to confirm the limits were there, he wasn't able to get them raised or removed, nor could he tell us the limits by machine type.
There's a thread from 18 months ago on the aws forums [1], where an aws rep more or less confirms, but again provides no information
[1] https://forums.aws.amazon.com/thread.jspa?threadID=231806
The "Connection Tracking" portion of http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-net... provides some insight here and describes a method to avoid connection tracking in EC2's firewall.
Ugh, I hate that. They also have hidden limits on the number of incoming tcp connections you can have.