The folks behind audio set have been working on general audio event detection for some years now, I believe.

https://research.google.com/audioset/

There's a huge amount to discuss in the audio domain... But for a starting place, using ResNet on spectrograms to build a binary classifier is a good place to start.