It's way way way bigger than the number that have custom built ultra-high end PCs.

Phones that will be powerful enough to provide a good experience for more than a short amount of time (due to battery, etc.) and with the abilities of other VR systems (hand controllers, positional tracking, etc) are going to be a fair ways off.

Yes all gamers (including phones) is a bigger group than console gamers, but there are still a TON of console gaming households who have proven willing to throw down real money to buy hardware to play specific games.

It remains to be seen how many of these will buy a VR headset. Peripherals have always been a very marginal aspect even on dedicated gaming hardware. If there's an actual killer application (i.e. a must have game) I could see myself buying into it, but as long as the VR "games" are either glorified demos or badly converted regular games that are worse than their non-VR counterpart, I won't spend a cent on it.

With some exceptions like Game of War, Candy crush and Pokemon Go the average mobile user doesn't play that much. They just need a quick 5 minute distraction. That's the wrong mindset for VR.

I'm not a big fan of "console/PC or it's fake" attitudes towards games. Some of the best games I've played in years were on iPhone/iPad.