Re:600K concurrent HTTP connections with Clojure and http-kit
by javajosh on Monday January 28, 2013 @01:59AM
(#5127520)
Running locally doesn't mean you bypass TCP/IP.
[ Parent
| on HN ]
Re:600K concurrent HTTP connections with Clojure and http-kit
by mbell on Monday January 28, 2013 @03:54AM
(#5127729)
While true, I'm suspect given this setup that some layer isn't just dropping to a unix socket when it notices localhost.
Most notably, 600k active real TCP connections in the kernel would use somewhere around 6GB of memory assuming an average of ~10kB of memory per socket for the R/W buffers and other data. EDIT: Thats just the server side, double to include the client sockets.
Most attempts I've see at this number of real TCP connections required a lot more tweaking of kernel TCP settings to achieve.
Re:600K concurrent HTTP connections with Clojure and http-kit
by alexkus on Monday January 28, 2013 @05:44AM
(#5127980)
For large number (1M+) of connections (doing websockets or long polling) to replicate the functionality of a COMET server I've been playing with rolling my own TCP handling via libnetfilter_queue as I simple don't need the ~10kB of r/w buffers on each socket.
Linux (as far as I'm aware) doesn't allow tuning of r/w buffer sizes on a per interface basis otherwise I'd have one interface for the COMET server with drastically reduced r/w buffer sizes and the remaining interfaces with 'normal' TCP r/w buffer sizes to ensure the other things running on that host run without problem.
Re:600K concurrent HTTP connections with Clojure and http-kit
by justincormack on Monday January 28, 2013 @03:57AM
(#5127735)
He has 16G of RAM so may have not tuned this, and not noticed.
Re:600K concurrent HTTP connections with Clojure and http-kit
by justincormack on Monday January 28, 2013 @03:55AM
(#5127732)
It bypasses checksumming, but no not the whole stack. Although not sure in this case it bypasses checksumming as the interfaces were created as aliases on a physical interface, which may mean it does do checksumming still.
Re:600K concurrent HTTP connections with Clojure and http-kit
by voidlogic on Monday January 28, 2013 @03:52PM
(#5130564)
Many OS's kernels use their local domain socket codepath/interface for 127.0.0.1 <-> 127.0.0.1 communications, completely bypassing TCP/IP... This is transparent to the application.