Level-wise they are separate though. It should be easy enough to find the proper gain for the reference recording (how loud does it sound in the video?), and then subtracted, leaving any residual energy in that band from desired sources. Yeah, there will be some gain-matching and spatial artifacts, but probably not enough to make things (voices for instance) unintelligible.

Thats actually where any ML component would naturally sit. It would try to basically determine a "close-enough" deconvolution envelope for the reference recording.