Actually, you don't even need xvfb these days (and I have no idea why anyone is telling people that they do). You could use something like xserver-xorg-video-dummy. Of course, as you have pointed out, PhantomJS doesn't require an X server (as opposed to these other projects, wtf) so that it is "really really truly headless".