Going to go with drivers again. The sad truth is closed source drivers can be as hit or miss as open source. By any measure I have tried, Windows 7 is considerably faster then XP for the gear I run, but only when proper drivers are installed (not the included Windows 7 ones).
It is annoying when you get something that's slower or doesn't work though. I was burnt by Creative with my Live! Drive and as such will never buy their hardware or software again, but I don't blame Microsoft.
I don't feel the speed difference between an Ubuntu/XP is as big as in between Ubuntu/win7 (talking about the old-desktop here).