News for hackers. Stuff that matters.
Re:Maintaining 65k open connections in a single Ruby process by jsnell on Monday October 29, 2018 @06:23PM (#18332135)
> The maximum amount of connections achieved was 65523, just 13 short of 2^16. This is the theoretical maximum number of connections for a single IP address on a single port,

That's not right. Connections are identified by the 4-tuple of (source ip, source port, destination ip, destination port). The server ip and port are fixed in this example, but there's still 2^48 combinations of client ip/port.

> It is amazing that a problem which took huge engineering efforts to solve back in the early 2000s can now be solved easily by anyone in just a handful lines of code.

But you didn't solve the problem! The whole point of C10K was the need for scalable ways to figure out which connections could be operated on (i.e. had something to receive, or could be written to after having previously been blocked for writes). Keeping an array of sockets and just repeatedly iterating over it with the assumption that all those sockets are active is the opposite of scalable.

Re:Maintaining 65k open connections in a single Ruby process by WJW on Monday October 29, 2018 @06:35PM (#18332237)
Can you explain more how you get to 2^48? I thought the protocol specification only reserved two bytes for the port and so getting over 65536 connections would either require using more IP adresses or more ports. This seemed supported by answers like https://www.quora.com/What-is-the-maximum-number-of-concurre.... If you can enlighten me, I'd be much obliged.

You are definitely right that this is not a "full" solution for the c10k problem, but it is an interesting start IMO. Maybe for next month I'll implement basic epoll functionality to make an echo server or something :)

Re:Maintaining 65k open connections in a single Ruby process by toast0 on Monday October 29, 2018 @06:57PM (#18332408)
If I'm a server, and my ip is 192.0.2.1, and i listen on port 80, my nearby clients 192.0.2.2 and 192.0.2.3 can each establish just about 64k connections to me (port 0 doesn't generally work). I like to think of it as a 5-tuple

{Protocol, PeerA, PeerB, PortA, PortB} ex:

{tcp, 192.0.2.1, 192.0.2.2, 80, 12345} is distinct from {tcp, 192.0.2.1, 192.0.2.3, 80, 12345}

With roughly 2^32 ipv4 addresses and 2^16 client ports for each ip, you should be able to establish 2^48 connections to your server IP. Although, it usually requires a bit more than standard effort to actually use quite that many client ports per IP, by default you have to fight against high vs low ports, and anyway not a lot of OSes have a very sane allocation policy for ports.

Note also: if you're testing this on localhost, IANA has assigned you 2^24 ipv4 addresses for localhost, 127.0.0.0/8, so if i assume you're going to listen on all addresses, and connect to all addresses from all addresses, that gives you 2^24 x ( 2^24 x 2^16 ) = 2^64 connections. It might take you some time to iterate through an array that large :)

Re:Maintaining 65k open connections in a single Ruby process by masklinn on Monday October 29, 2018 @06:36PM (#18332239)
> Can you explain more how you get to 2^48? I thought the protocol specification only reserved two bytes for the port and so getting over 65536 connections would either require using more IP adresses or more ports.

32b IP * 16 bit port?

> You are definitely right that this is not a "full" solution for the c10k problem, but it is an interesting start IMO.

It's not a solution to the C10K at all, because the C10K problem is outdated and was trivialised a decade back. Whatsapp was doing 2 million connections on a single box (with resources to spare) back in 2012: https://blog.whatsapp.com/196/1-million-is-so-2011 and folks were working on C10M: http://highscalability.com/blog/2013/5/13/the-secret-to-10-m...

> Maybe for next month I'll implement basic epoll functionality to make an echo server or something :)

Epoll is bad, why would you want to do that?

Re:Maintaining 65k open connections in a single Ruby process by meguest on Monday October 29, 2018 @08:18PM (#18332914)
> epoll is bad

Source please! I'm genuine - what is bad about it?

Re:Maintaining 65k open connections in a single Ruby process by masklinn on Tuesday October 30, 2018 @05:04AM (#18334890)
Re:Maintaining 65k open connections in a single Ruby process by kjeetgill on Tuesday October 30, 2018 @01:24AM (#18334155)
It comes up all the time but I'm not qualified to speak to it off the top of my head.

Google comparisons with kqueue from BSD and iocp on Windows.

Re:Maintaining 65k open connections in a single Ruby process by john37386 on Monday October 29, 2018 @06:33PM (#18332219)
I'm sure that your ruby can hold +100k connctions with enough RAM.
Re:Maintaining 65k open connections in a single Ruby process by WJW on Monday October 29, 2018 @06:39PM (#18332270)
I ran 4 `ab` processes against it, each with 20k concurrent connections, so the maximum amount of connections "should" have been 80k. The fact that the actual number was so close to 65536 makes me very suspicious that there is some kernel level thing at play here. I agree that the ruby program can probably scale up much more with enough RAM.
Re:Maintaining 65k open connections in a single Ruby process by andrewmackrodt on Tuesday October 30, 2018 @12:21PM (#18338272)
Would this have been affected by the number of available ports on your single loopback address, assuming 127.0.0.1. You'd have to connect via multiple network devices.

Edit: comment below clarifies servers.

Re:Maintaining 65k open connections in a single Ruby process by john37386 on Monday October 29, 2018 @06:45PM (#18332330)
Was iptables running? Maybe they hit a conntrack limit
Re:Maintaining 65k open connections in a single Ruby process by tyingq on Monday October 29, 2018 @06:51PM (#18332373)
What does ulimit -a say?
Re:Maintaining 65k open connections in a single Ruby process by tyingq on Monday October 29, 2018 @06:39PM (#18332268)
It does make you wonder what was throttling it to 65k. I didn't see any mention of tweaking kernel/ulimit/systemd max open file settings.
Re:Maintaining 65k open connections in a single Ruby process by masklinn on Monday October 29, 2018 @06:43PM (#18332317)
Since client connecting to a single port? So you'd get 1 client IP, 65k client ports, connecting to 1 destination IP on 1 port.
Re:Maintaining 65k open connections in a single Ruby process by tyingq on Monday October 29, 2018 @06:44PM (#18332325)
"Eventually we got it working by using four servers each running a single instance of ab in addition to the one running our Ruby process"

So 4 source ips. Guessing they were just bumping into a (configurable) max open file setting.