Interesting project. In my experience as a re-recording mixer for film the best denoisers currently on the market work as a sort of spectral noise gates: so instead of running one noise gate over the whole audio spectrum, you split it into frequency bands each with their own thresholds. These thresholds are tuned in by “learning” the spectral shape of the background noise itself and usually the user can still modify the wheights with a curve (e.g. if you want to gate high frequencies more).

This has the benefit of producing more natural results while keeping speech understandable. It might interest you, because with a 2000x speed marging you could still crank the algorithm up.