PHK has argued that the SSL/TLS libraries we have is a nightmare when it comes security design. He rather favor "jailing" these processes as a separate process. So if hell breaks lose, like with Heartbleed, your Varnish process should still be "safe".
If you're serving HTTPS to clients and your backends are also HTTPS, you quickly get into a mess of extra components opening unnecessary sockets around just to do TLS.