Sounds like you had a gigantic malloc and used too much memory and let Linux kill it.

http://stackoverflow.com/questions/16693677/why-isnt-varnish...

If you really need a large malloc, I highly recommend running Varnish on FreeBSD instead of Linux as the Linux kernels VM system is horrible when memory is overcommitted.