Something is definitely wrong with your setup. You should not experience any noticeable tracking errors on current hardware.
Some combination of a bug, strange setup, or bad environment is going on here. Maybe you have a large mirror or pane of glass around your setup? Lots of things can go wrong.
Echo VR is super super fun to play, as one concrete example, but I’d say the Rift is still not good enough other than as a toy to show off to people. The driver updates are also absurdly huge. I didn’t use it for a month and I had a 14gb driver download (pretty sure it wasn’t any game/app updates)
We are far away from VR being even in the comparable to general purpose “everyday” computing IMO.