News for hackers. Stuff that matters.

A comment on One Million Concurrent TCP connections.

Re:One Million Concurrent TCP connections by mrb on Thursday September 22, 2011 @11:41PM (#3028741)
I remember reading around 2002-2004 about a sysadmin managing a very large "supernode" p2p server who was able to fine-tune its Linux kernel, and to recompile an optimized version of the p2p app (to allocate data structures as small as possible for each client) to support up to one million concurrent TCP connections. It wasn't a test system, it was a production server routinely reaching this many connection at its daily peak.

If it was possible in 2002-2004, I am not impressed that it is still possible in 2011.

One of the optimizations was to reduce the per-connection TCP buffers (net.ipv4.tcp_{mem,rmem,wmem}) to only allocate one physical memory page (4kB) per client, so that one million concurrent TCP connections would only need 4GB RAM. His machine had barely more than 4GB RAM (6 or 8? can't remember), which was a lot of RAM at the time.

I cannot find a link to my story though...

[ Parent | on HN ]
Re:One Million Concurrent TCP connections by gtani on Friday September 23, 2011 @12:50AM (#3028866)
(Not your link but) some pointers on tuning erlang/OTP servers:

http://www.erlang-consulting.com/thesis/tcp_optimisation/tcp...

http://www.trapexit.org/Building_a_Non-blocking_TCP_server_u...

http://groups.google.com/group/erlang-programming/browse_thr...

(this erl mailing list thread is pretty typical, if you put up code, describe your app, hardware, network, database/external dependencies, etc, you'll get a ton of good advice about killing off bottlenecks. Another example

http://groups.google.com/group/erlang-programming/browse_frm...

[ Parent | on HN ]
Re:One Million Concurrent TCP connections by mmaunder on Friday September 23, 2011 @02:58AM (#3029124)
Agree with this being underwhelming. Maybe they'll hire someone who understands that Erlang is using kqueue to make this possible.

Running netstat|grep like this on a high concurrency server takes a long time to run. I've never found a faster way to get real-time stats on our busy servers and would be interested if anyone else has.

[ Parent | on HN ]
Re:One Million Concurrent TCP connections by thomasknowles on Friday September 23, 2011 @04:18AM (#3029258)
Actually I would run ss over the netstat for such high concurrency connection as netstat hits /proc/net/tcp and it's quite slow. However, you need to have the tcp_diag module loaded otherwise it falls back to /proc/net/tcp.
[ Parent | on HN ]
Re:One Million Concurrent TCP connections by X-Istence on Friday September 23, 2011 @11:50AM (#3030397)
You may not have read the entire article, but he clearly explains that their tech stack is FreeBSD + Erlang not Linux + Erlang.

There is no /proc/net/tcp on FreeBSD. Hell, there is no /proc unless it is specifically mounted by the administrator, but tools certainly don't use it to get data.

[ Parent | on HN ]
Re:One Million Concurrent TCP connections by thomasknowles on Saturday September 24, 2011 @06:51AM (#3032968)
I read it, I just presume people most users are linux based in here.
[ Parent | on HN ]
Re:One Million Concurrent TCP connections by derobert on Friday September 23, 2011 @03:23AM (#3029167)
You might be able to parse /proc/net/tcp yourself faster, especially for just counting by state.
[ Parent | on HN ]
Re:One Million Concurrent TCP connections by X-Istence on Friday September 23, 2011 @11:51AM (#3030399)
/proc/net/tcp doesn't exist on FreeBSD...
[ Parent | on HN ]
Re:One Million Concurrent TCP connections by adient on Friday September 23, 2011 @07:16AM (#3029589)
You may have tried but generally add -n to netstat to speed it up by just displaying IPs instead of trying to resolve them. I've never seen a system with so many concurrent connections hands on so there could be some other bottleneck.
[ Parent | on HN ]
Re:One Million Concurrent TCP connections by jen_h on Friday September 23, 2011 @09:30AM (#3029888)
netstat -s |grep "connections established" is pretty fast and does the job if you're not filtering for certain ports or IPs. But running plain old netstat on a system with so many connections is a really bad idea. I almost fired myself once over a crufty cron job I left running...
[ Parent | on HN ]