I think this is something that neural networks would be great at, I suppose the hard part is figuring out exactly what to train one for/with.

As a starting point, you could separate the different instruments, which I'm pretty sure is a solved problem by now.