Servers can have up to 1TB of ram without becoming overpriced.
But starving 80k users due to low buffering will be expensive in the long run. Way more expensive than RAM ;-)
Still, 200kB is excessive. A program with buffers no larger than 10kB can _easily_ saturate a 1 Gbit/s NIC. Hitch is designed to handle many concurrent connections, so even if it handled a paltry 10 connections it could easily saturate a 10 Gbit/s NIC with 10kB buffers. If not, then there is a design flaw somewhere.
"Servers can have up to 1TB of ram without becoming overpriced"
This is irrelevant. If a Hitch version had a default overhead of 10kB per connection, it could in theory scale to 20x the number of connections than this version of Hitch, for a given amount of RAM (no matter the amount). Maximizing the use you get out of a given amount of hardware resources should be your priority when writing scalable software.
Let me ask a leading question: how much of this do you think is openssl overhead?
Please consider optimising for a real usage scenario, not some fantasy benchmarking setup.
The minimum theoretical memory usage is 4kB per connection: 1 page in kernel space, and nothing on the userland (eg. you use zero-copy to txfer data between sockets or to/from file descriptors).
At Google, our SSL/TLS overhead per connection is 10kB: https://www.imperialviolet.org/2010/06/25/overclocking-ssl.h...
Secure sockets have a lot more overhead than plain TCP sockets, on top of that it has all of the overhead that a proxy has per connection.
You should also set SSL_MODE_RELEASE_BUFFERS to reclaim memory from idle SSL connections.
Ouch. This default means 16GB only lets you handle 80k connections. I would hardly call this "scalable". In 2015 you find blog posts left and right showing you how to reach 1 million connections on a single machine with language X or tool Y or framework Z. Maybe the developers should change this default.
https://news.ycombinator.com/item?id=3028741