News for hackers. Stuff that matters.
Re:600k concurrent websocket connections on AWS using Node.j... by sciurus on Friday October 11, 2019 @09:11AM (#21223981)
A few pieces of advice based on running https://github.com/mozilla-services/autopush-rs, which handles tens of millions of concurrent connections across a fleet of small EC2 instances.

1) Consider not running the largest instance you need to handle your workload, but instead distributing it across smaller instances. This allows for progressive rollout to test new versions, reduces the thundering herd when you restart or replace an instance, etc.

2) Don't set up security group rules that limit what addresses can connect to your websocket port. As soon as you do that connection tracking kicks in and you'll hit undocumented hard limits on the number of established connections to an instance. These limits vary based on the instance size and can easily become your bottleneck.

3) Beware of ELBs. Under the hood an ELB is made of multiple load balancers and is supposed to scale out when those load balancers hit capacity. A single load balancer can only handle a certain number of concurrent connections. In my experience ELBs don't automatically scale our when that limit is reached. You need AWS support to manually do that for you. At a certain traffic level, expect support to tell you to create multiple ELBs and distribute traffic across them yourself. ALBs or NLBs may handle this better; I'm not sure. If possible design your system to distribute connections itself instead of requiring a load balancer.

2 and 3 are frustrating because they happen at a layer of EC2 that you have little visibility into. The best way to avoid problems is to test everything at the expected real user load. In our case, when we were planning a change that would dramatically increase the number of clients connecting and doing real work, we first used our experimentation system to have a set of clients establish a dummy connection, then gradually ramped up that number of clients in the experiment as we worked through issues.

Re:600k concurrent websocket connections on AWS using Node.j... by Thaxll on Friday October 11, 2019 @09:45AM (#21224268)
If you have a lot of trafic you shoudn't use an ELB in the first place, should be using an NLB by default it comes with a 40Gb pipe and multi millions connections.

https://docs.aws.amazon.com/elasticloadbalancing/latest/netw...

Re:600k concurrent websocket connections on AWS using Node.j... by windlep on Friday October 11, 2019 @11:35AM (#21225369)
We did not use an NLB because it is expensive in its pricing model for TLS connections (and we prefer to let AWS terminate TLS). For 20 million connections, an NLB would cost $28,800/month. It's drastically cheaper to use an ELB, provided you can talk to AWS to have them scale it for you.
Re:600k concurrent websocket connections on AWS using Node.j... by mnutt on Friday October 11, 2019 @04:21PM (#21228095)
When I last discussed this with AWS, having them scale it for you meant that you actually had to contact them to pre-warm the ELB whenever you expected more traffic. It didn't work for our use cases where we didn't know when to expect more traffic.
Re:600k concurrent websocket connections on AWS using Node.j... by windlep on Saturday October 12, 2019 @12:10AM (#21230616)
Yup, that hasn't changed in our experience. If you're looking at bursty traffic beyond what AWS will pre-warm for you, then this likely won't work. We have fairly consistent loads throughout the day.
Re:600k concurrent websocket connections on AWS using Node.j... by papito on Friday October 11, 2019 @02:27PM (#21227027)
Back in the day, in my first job, I created load balancing using LVS. It was not easy to set up, compared to AWS, but it worked, and it was completely free, plus some man-hours up front. I wonder if it would be worth it nowadays...
Re:600k concurrent websocket connections on AWS using Node.j... by xyzzy_plugh on Friday October 11, 2019 @03:36PM (#21227678)
Is it drastically cheaper to use an NLB without letting AWS terminate TLS? I thought AWS was expensive for TLS termination, generally, unless you need the client locality for lower latency.
Re:600k concurrent websocket connections on AWS using Node.j... by windlep on Saturday October 12, 2019 @12:08AM (#21230611)
Yes, if you terminate TLS its drastically cheaper. Without TLS, each NLB unit lets you have 100,000 concurrent connections. With TLS, a single unit is 3,000 connections. Similar pricing occurs with ALB's.

Terminating TLS yourself incurs some CPU cost and a bit more memory cost. How much CPU/memory is eaten depends on the efficiency of your code. Our Rust implementation roughly matches C code efficiency, so we could handle terminating TLS ourselves if ELB stops being feasible at some point.

Re:600k concurrent websocket connections on AWS using Node.j... by arrty88 on Friday October 11, 2019 @09:59AM (#21224403)
You can also employ DNS round robin with health checks in Route53
Re:600k concurrent websocket connections on AWS using Node.j... by Thaxll on Friday October 11, 2019 @10:13AM (#21224533)
ELB is pretty old and should be avoided tbh, use ALB and if you can't should switch to NLB ( for pure tcp LB ).
Re:600k concurrent websocket connections on AWS using Node.j... by windlep on Friday October 11, 2019 @12:05PM (#21225669)
Same issue as NLB, they charge for units based on concurrent connections and it gets very expensive very quickly.
Re:600k concurrent websocket connections on AWS using Node.j... by song on Friday October 11, 2019 @01:03PM (#21226234)
DNS round robin has the disadvantage of not guaranteeing that the traffic will be equally distributed. This can be a problem if there's a lot of automated traffic.

But, with that said, in term of price, it's unbeatable.

Re:600k concurrent websocket connections on AWS using Node.j... by arrty88 on Saturday October 12, 2019 @11:30AM (#21233892)
I have had success DNS round robin-ing a cluster of HAproxy LBs which handle SSL and then proxy to my backend websocket gateways which keeps the real load evenly distributed where it matters and also run health checks on the nodes.
Re:600k concurrent websocket connections on AWS using Node.j... by sieabahlpark on Friday October 11, 2019 @02:34PM (#21227083)
The point is at scale you can't guarantee equal but something around there
Re:600k concurrent websocket connections on AWS using Node.j... by song on Friday October 11, 2019 @11:21PM (#21230429)
That works well, but if you do ecommerce or a web app for which people write a lot of bots, then the traffic won't necessarily be that well distributed (mostly because users while run bots out of dedicated servers or such and will tend to only query dns once).

A bit of an edge case but pretty much the only issue I've had so far which is why I bring it up.

Re:600k concurrent websocket connections on AWS using Node.j... by nathancahill on Friday October 11, 2019 @10:10AM (#21224513)
That's the first tool I grab for balancing _a lot_ of connections.
Re:600k concurrent websocket connections on AWS using Node.j... by gravypod on Friday October 11, 2019 @10:39AM (#21224789)
> 1) Consider not running the largest instance you need to handle your workload, but instead distributing it across smaller instances. This allows for progressive rollout to test new versions, reduces the thundering herd when you restart or replace an instance, etc.

I came here to say this. Horizontal compute is a miracle.

Re:600k concurrent websocket connections on AWS using Node.j... by verttii on Friday October 11, 2019 @12:07PM (#21225681)
Distribution itself might be a world of pain depending on the nature of your application. For example stateful apps where low latency is expected.
Re:600k concurrent websocket connections on AWS using Node.j... by semiotagonal on Friday October 11, 2019 @01:11PM (#21226323)
How meaningful is the benchmark for handling idle connections? Handling more is better, of course, but if the server melts down when 0.1% of them have any activity, maybe maximizing idle connections isn't the right place to expend optimization effort?
Re:600k concurrent websocket connections on AWS using Node.j... by sciurus on Friday October 11, 2019 @03:35PM (#21227671)
For us, the experiment (https://blog.mozilla.org/services/2018/10/03/upcoming-push-s...) with mostly idle clients was very helpful, since it flushed out problems with out load balancing layer and in the end gave us confidence that it, our app, and our persistence layer could safely handle many more connections.

If we had begun having problems once we started sending more push messages, we would have simply stopped using the new service (https://github.com/mozilla-services/megaphone) responsible for that until we worked through them.

Re:600k concurrent websocket connections on AWS using Node.j... by cortesoft on Friday October 11, 2019 @02:06PM (#21226852)
Depends on your workload
Re:600k concurrent websocket connections on AWS using Node.j... by dylanz on Friday October 11, 2019 @09:16AM (#21224011)
Could you explain or post a link to something about #2? I’ve never heard of that before!
Re:600k concurrent websocket connections on AWS using Node.j... by sciurus on Friday October 11, 2019 @10:00AM (#21224414)
I can not. AWS, for whatever reason, does not want this publicly documented. I would write up what we learned in our testing in more detail, but I have been asked not to.

You can find some discussion of this behavior in places like https://forums.aws.amazon.com/thread.jspa?threadID=231806. I originally became aware of the issue, before hitting in production, from the HN comment at https://news.ycombinator.com/item?id=18314138

Re:600k concurrent websocket connections on AWS using Node.j... by toast0 on Friday October 11, 2019 @02:22PM (#21226992)
I'm glad you found my comment helpful!

I myself found out about to avoid the connection tracking in this thread https://news.ycombinator.com/item?id=15724072

It's super frustrating that Amazon doesn't document this in a more approachable manner.

Re:600k concurrent websocket connections on AWS using Node.j... by QuinnyPig on Friday October 11, 2019 @11:29PM (#21230460)
Their response displeases me.
Re:600k concurrent websocket connections on AWS using Node.j... by toredash on Saturday October 12, 2019 @10:40AM (#21233490)
This is what you need to know: "Not all flows of traffic are tracked. If a security group rule permits TCP or UDP flows for all traffic (0.0.0.0/0) and there is a corresponding rule in the other direction that permits all response traffic (0.0.0.0/0) for all ports (0-65535), then that flow of traffic is not tracked. The response traffic is therefore allowed to flow based on the inbound or outbound rule that permits the response traffic, and not on tracking information."
Re:600k concurrent websocket connections on AWS using Node.j... by chatmasta on Saturday October 12, 2019 @11:17AM (#21233791)
Very interesting. It makes a lot of sense from a Linux perspective. The conntrack table requires memory, so there is some physical limit on its size. That also explains why it scales per instance type.
Re:600k concurrent websocket connections on AWS using Node.j... by toredash on Saturday October 12, 2019 @12:02PM (#21234140)
This is also why "endless scaling" is a fud. At some point or some level, there is either a physical, soft, standard or some kind of limit
Re:600k concurrent websocket connections on AWS using Node.j... by runT1ME on Friday October 11, 2019 @04:27PM (#21228144)
I can corroborate this, at Verizon we had problems scaling out ELBs for a very similar use case.
Re:600k concurrent websocket connections on AWS using Node.j... by thecopy on Friday October 11, 2019 @09:41AM (#21224235)
I would also be interested in this. First time I hear about such a limitation.
Re:600k concurrent websocket connections on AWS using Node.j... by gchamonlive on Friday October 11, 2019 @09:19AM (#21224040)
I recently played around with Athena for load balancer logs, and with sql it is easy to cross reference many connection entries from the load balancer. Do you believe this could help in getting more visibility to spot bottleneck problems stated in 2 and 3?
Re:600k concurrent websocket connections on AWS using Node.j... by sciurus on Friday October 11, 2019 @10:47AM (#21224869)
Not really, because there won't be logs for connections that were never established.
Re:600k concurrent websocket connections on AWS using Node.j... by johne20 on Friday October 11, 2019 @11:07AM (#21225080)
How are you keeping track of those?
Re:600k concurrent websocket connections on AWS using Node.j... by sciurus on Friday October 11, 2019 @11:32AM (#21225328)
As I recall, when we hit the ELBs connection limits the failed connections were reflected in the SpilloverCount metric.

I believe we only hit the security group limits in load testing. There, we could see connections fail to an instance when that instance's established connections as reported by netstat or similar tools hit a certain threshold.