If you switch to nginx, then you probably won't need varnish at all and can replace Apache.
The nginx config is much, much, much easier to work with than Apache. Transferring complex rewrite rules is the hardest thing to get right when moving over, but everything else is much easier.
Smart lobsters use nginx
* Changing Apache would have required changing other packages - PHP wasn't thread-safe.
* Varnish looked fragile. I wish I still had my notes about why I thought that.
* Nginx in front of Apache was relatively quick to setup & really great in practice - restarting an order of magnitude less often.
If the nginx config file would be useful as a starting point, I offer it.
Edit: formatting.
I'm not sure if it would help me, but I'm almost certain it would help other people - so please, consider your offer accepted if you can find somewhere easy to publish this :)
(Compression & other varying content was a headache - separately caching & serving gzip, deflate, & unpacked versions, without any built-in compression.)
Those tweaks looked time-consuming to set up, hard to test, & liable to break quietly. A buffer in the way that simply & quickly gets a response from (& then frees up) Apache, then slowly feeds the response to the client, provides most of the benefit; nginx fills this role well for me.
Having said that, I'm looking at Varnish (once the stampede dies down) ...