You take an extra hit the first time a TLS session key is established, after that all subsequent HTTP requests can do a fast resume. This has nothing to do with users over a period of time.