Yeah, the combination of those two things is very likely to not do you any favors.
It is worth clarifying that Google et al.'s claim that SSL is essentially no overhead is conditioned on the assumption that you're using reasonably modern and full-featured processors, especially with AES-GCM in hardware. (Which is pretty common on laptop processors these days even without trying hard to find it, but probably won't be on an Atom HTPC.) I think that's reasonable, since if you're seriously worried about performance and latency, you're probably starting off with good hardware, and your worry is that investment will go to waste if you turn on SSL. At least for running a web server for fun on an old personal machine, the added latency is real and is unfortunate but I'd guess also not such a big deal. But maybe that's a bad assumption?
I'm also using 4Kbit RSA keys, maybe that's the cause, especially given that the server is a tiny Atom HTPC sitting in the kitchen (100ms is because I'm accessing it from the other country). Will try to find some time on weekend and test with 2Kbit ones to see if this is indeed the cause.
--------------
Added: seems that this worsens with latency, because I see extra 200ms. Maybe the cause is extra network round-trips, not crypto overhead. Or maybe there's something with my curl...
Unfortunately, don't have time to meditate on Wireshark output right now. :(