if your user goes to your site twice, he has your assets in cache.

It's not good practice to rely on a user's cache. If they visit your site twice in a span of a day, they may have already cleared their cache or pushed your files out with other sites' files.

This article demonstrates a pretty good method to reduce unnecessary byte transfer which will ultimately speed up page loading.

Caching is still a good idea, because if a user decides to browse to another page on the same site in the same visit (or refresh/post a comment), a significant fraction of the data required for it has already been loaded. If a user does come back, there's always a chance that your stuff will still be in the cache.

If your site loads slowly he won't come back the second time.