That seems solvable, for example by blocking programmatic control of playback until you allow it for a given video element. That way it only "breaks" when the user wants it to be "broken".