> If you were actually building a service like that in Go (millions of active connections) and you were very concerned about memory usage

Or, much more likely, pay a few tens of dollars per month for a second server and start scaling out.