This sort of exists, actually... use hover focus and eViacam (https://eviacam.crea-si.com/) together. Of course, that doesn't give you separate control of the mouse, but I'm not sure that's a bug.