Unless you are regularly using all your RAM, there is a very good chance the browser's files are going to stay in disk cache and be effectively instant to load. 100MB is an imperceptible load time difference on a modern system where memory speed is 50-100GB/s or more.
Personally, around 5 times per day, every day. When I don't browse, I close the browser