Maybe, but velocity is relative. So that would mean that the drive is pushing against something which has its own rest frame, relative to which velocity can be measured.

It's not clear what that rest frame would be. Virtual particles in the quantum vacuum are often mentioned together with the emDrive, but those particles are thought to be frameless.

I'll admit that it's not entirely implausible to have a special rest frame like that. We already know that the cosmic microwave background has a rest frame, and this is something that you would start to feel in the form of frontal radiation pressure as you gain speed relative to the CMB.