To break the attack in this paper, you don't even need a hardware RNG; you just need hardware AES, like most modern platforms have (and like most mainstream operating systems use by default).
Suddenly after I read what you typed about RDRAND it clicked to me - you must never fully trust the hardware. Even if you TRUST the HW RNG, what is the harm of combining it into a broader RNG (assuming you know what you are doing).
Thanks for your time.
With hardware you have to always worry about some "physical" error, and if the hardware doesn't have sufficient diagnostics, you will never find out in software, because after whitening even a constant zero series from hardware looks indistinguishable from the failure-free output.
Intel itself ships expensive accelerator cards that have "zero software" in key seed generation capable of generating multiple gigabytes per second of random bits.
Slightly off-topic: Would it be near-impossible to have a hardware-level RNG generator that spits out bits at a sufficient enough rate to avoid software-based RNG schemes? My thought is to have a very-very-vetted hardware RNG, and use that as an anchor to build off of.