You should have a talk with your adviser and discuss your options. If you don't feel comfortable talking with him about this, do you really want him as an adviser for the next several years?

If you really want to work with him, I would follow him. As a condition for moving to another university, I'm sure he could stipulate that he be allowed to bring in some people.

Perhaps you can just be a research assistant for a year.