It would probably not be possible to get away with an XOR instruction that doesn't behave as advertised. There is simply no way to determine if RDRAND is behaving as advertised or not.
> you have to trust someone
Well, yes. Intel specifically has to my mind demonstrated beyond a shadow of a doubt that they are not be trusted (that "entire x86 core running minix --- with security holes" thing you just mentioned, thought it was ARM though). So if the only difference is moving my root of trust from Intel to a random, unknown party that is not Intel, that is already something of an improvement.
The main difference is that the behavior of the rest of the CPU is deterministic, so it can be verified. When you use the ADD instruction, the output will always be the sum of its inputs; when you use the RDRAND instruction, the output comes from a black box which is supposed to mix internal non-deterministic noise sources in a non-reversible way. How can you distinguish a correct RDRAND from a malicious or broken one which generates its output in a reversible way or based on a fixed key, since the apparent output (an arbitrary number) is the same in both cases?
[1]: https://techreport.com/review/17732/intel-graphics-drivers-e...
The claim is not "can't trust RDRAND [to keep your system secure] because could be backdoored", the claim is "can't trust RDRAND [to generate unique unguessable numbers] because could be backdoored.
RDRAND not only can be backdoored, but its implementation flawed or too weak for some attackers. Or it can be a pinnacle of RNGs(unlikely). In any case, it is reasonable to not trust it to generate unique unguessable numbers. That trust requires control of the generation.
The impact this distrust has on decision of using Intel CPU's is specific to the use case. I trust my Intel CPU to do deterministic computational work for me, because in some cases I can rerun the job and compare the results, or check the results in another way. But I don't trust it to generate unguessable numbers for me. That is better done by separate HW device, ideally one that I would construct and control.
I don't have to trust implicitly in my computer to use it. At the same time, I do not want my kernel to rely on RDRAND to get random numbers, if more trustworthy source is viable. No that perfect randomness is required, but if the authors can do better than rely on RDRAND, they should.
> The bottom line is the entire CPU can not be audited. So trust it. Or don't trust it.
I can trust it to evaluate 1234+1234, or play a video for me, but I don't trust it to generate unique unguessable numbers.
There is trusting them not to be malicious.
Then there is trusting them not to more innocently have some errata effecting quality of randomness.
Do we? I was under the impression that most people would not care if the bootloader did that or not.
Intel hid an entire x86 core running minix --- with security holes --- and told no one. Between the firmware code which runs in System Management Mode and in UEFI, which persists after the OS is booted and can take over the system and read and write arbitrary memory locations --- if you fear a backdoor in the CPU, RDRAND is not the only place where an attacker can screw you over. The bottom line is the entire CPU can not be audited. So trust it. Or don't trust it. Use another CPU architecture. Or go back to using pen and paper, and don't use any computers or cell phones at all.
We have to trust the bootloader to verify the digital signature on the kernel, so we might as well trust the bootloader to get a secure random seed. But from where? It could call UEFI or try use the RNG from the TPM (if available). But now you have to trust Intel, and/or the motherboard manufacturer, and/or the TPM. The bootloader could read from a seed file from the HDD/SSD. But we know that nation-state intelligence agencies (like the NSA) have the capability of implanting malware into HDD firmware (which is also unauditable), and we also know that we can't trust most consumer grade manufacturers or IOT devices to correctly insert device-specific secure entropy into the seed file at manufacturing time. Otherwise, all of the seed files will likely have the same value, at which point when the device is generating its long-term private key, immediately after it is first plugged in, the key will very likely be weak (see the "Mining your p's and q's"[1] paper).
The bottom line is that unless you propose to personally wire up your CPU from transistors, and then create your own compiler from assembly language (see the "Reflections on Trusting Trust"[2] paper by Ken Thompson about what can be hidden inside an untrustworthy compiler), and then personally audit all of the open source code you propose to compile using that compiler and use on your system, you have to trust someone.
What makes this political, and difficult to address, is everyone has different opinions on what they are willing to trust and not trust. But some combinations, such as "we can't trust Intel because their firmware can't be audited", and "but we insist on using Intel CPU's", really don't make much sense.
[1] https://factorable.net
[2] https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p7...