Thus there are no small objects scattered around--- each 'object' is contiguous, and most likely on a single page. Prefetches happen automatically --- in Linux at least, disk caching and the VM are essentially synonymous. The benefit of using the VM directly with mmap() is that you have more control over the details and less overhead.
Ah that could explain it. The people I know seem to be working on fs-level caches that operate at least in part on file granularity, so maybe they assume Linux/FBSD do fs-level stuff as well. They seem to try to do things like deciding whether to cache or not based in part on how big the file being read is, and what its historical usage patterns are.
By only having the one file so the number of system calls for a read or a write is '1', no need to juggle file descriptors (which you really really need for your sockets, not for the file system in most varnish setups).
Having 'small' objects clustered together is a good idea anyway, but you can do that in a big file just as easily as you could do it in multiple smaller files.
That's all hearsay, though, and I have no idea if it actually improves things in practice on current OSs or with which kinds of workloads.