Re:Ask HN: Is there an ML model that can go from an audio so...
by crtified on Wednesday June 07, 2023 @06:32PM
(#36234583)
That is insane(ly cool). Do you know how the separation models work, in principle?
Re:Ask HN: Is there an ML model that can go from an audio so...
by ikhatri on Wednesday June 07, 2023 @06:49PM
(#36234753)
Yup! It's a U-Net style convolutional neural network that runs on the spectrograms. There's a short 2 page abstract here: https://archives.ismir.net/ismir2019/latebreaking/000036.pdf and they cite this prior work as where they got the architecture from: https://arxiv.org/abs/1906.02618
In order to do this you can use a source/stem separation model like spleeter (https://github.com/deezer/spleeter) and then run the basic pitch model (or any other midi transcription model). There's other you can try which may yield better results, for example: (https://github.com/Music-and-Culture-Technology-Lab/omnizart)
Either way the key words you want to be looking for are "midi transcription" and "stem separation", should help you find more models to try for both steps. Good luck! :)