The software approach generally has lousy performance due to the extra overhead of forwarding rendering commands between the guest and host. In addition, since this approach requires writing a completely new graphics driver for the guest, there tend to be limitations with rendering API compatibility, as well as bugs. With the hardware approach, the guest can use the GPU vendor’s normal driver.
Virtualbox guest didn't participate in the contest, as I find graphical performance of VBox guests noticeably worse.
I must add though, that for desktop tasks VMWare Workstation performance is absolutely adequate, able to run double 2K monitors wonderfully. For gaming setup I'm personally planning to play with PCI-passthrough. The only problem is not to get lost in the tinkering, before playing any actual games :)
I remember even SketchUp wouldn't start in VirtualBox. I was able to get SketchUp running in VMWare Player, but the performance was very bad.
edit: this was also about 1.5 years ago, not sure how much improvements VMWare and VirtualBox have made in that time in terms of virtualizing GPU calls.
That is why I decided to end up setting up a VM with GPU passthru. It works extremely well for my usecase. I was able to score a cheap GPU for my Linux host (since I do not do anything graphically intensive on Linux, just need to drive 3 monitors), and was able to passthru my more expensive GPU over to the Windows VM. I would assume if you have integrated graphics on your CPU, you could use that for your host OS and get away with having only a single GPU.
But looking at the online docs for VMWare Workstation, it sounds like they take a different approach: the guest VM has a virtual device driver that lets both host and guest OSs use the same GPU at the same time.
I would think the VMWare Workstation-like approach is preferred, because it avoids the hassle of using two graphics cards. I know that Workstation is expensive ($250) and not OSS. Besides those two reasons, are there still reasons to prefer the PCI-passthrough approach?