Oh no, I thought of that. As a matter of fact I'm currently developing a VR game that would work just fine without controllers.
Still, sometimes you need buttons. While mobile gaming is obviously huge there are certain types of games that just don't work all that well on it. In the gaming space Apple will be directly competing against other VR headsets designed for gaming. Mobile games don't really directly compete with consoles or PCs.
Yeah, it will be limiting for the preexisting games built for preexisting VR systems. It will also be perfect for the types of games people envision and build for this generation of device. Not recognizing that seems more like a lack of imagination on your part than a failure of design on Apple’s.