Audio AI: isolating vocals from stereo music using Convolutional Neural Networks

Re:Audio AI: isolating vocals from stereo music using Convol... by btown on Friday February 15, 2019 @01:58PM (#19173657)

This is an awesome project, but it seems it was done without reference to academic literature on source separation. In fact, people have been doing audio source separation for years with neural networks.

For instance, Eric Humphrey at Spotify Music Understanding Group describes using a U-Net architecture here: https://medium.com/this-week-in-machine-learning-ai/separati... - paper at http://openaccess.city.ac.uk/19289/1/7bb8d1600fba70dd7940877...

They compare their performance to the widely-cited state of the art Chimera model (Luo 2017): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5791533/#R24 with examples at http://naplab.ee.columbia.edu/ivs.html - from the examples, there's significantly less distortion than OP.

Not to discourage OP from doing first-principles research at all! But it's often useful to engage with the larger community and know what's succeeded and failed in the past. This is a problem domain where progress could change the entire creative landscape around derivative works ("mashups" and the like), and interested researchers could do well to look towards collaboration rather than reinventing each others' wheels.

EDIT: The SANE conference has talks by Humphrey and many others available online: https://www.youtube.com/channel/UCsdxfneC1EdPorDUq9_XUJA/vid...

[ Parent | on HN ]

Re:Audio AI: isolating vocals from stereo music using Convol... by musicale on Friday February 15, 2019 @11:40PM (#19177102)

People have also been doing audio source separation effectively for years without neural networks.

[ Parent | on HN ]

Re:Audio AI: isolating vocals from stereo music using Convol... by calf on Saturday February 16, 2019 @12:49AM (#19177289)

It's interesting cause I have a recording of human voices plus a background TV show that was too loud; I've looked around for something that would be able to separate the two but I haven't found a straightforward solution.

For example if you Google then FASST is one of the ones that come up, but it's a whole framework and in order to use it you'd have to learn the research yourself; much of these software is not geared for end users.

[ Parent | on HN ]

Re:Audio AI: isolating vocals from stereo music using Convol... by jdietrich on Saturday February 16, 2019 @03:10AM (#19177659)

iZotope RX.

https://www.izotope.com/en/products/repair-and-edit/rx.html

[ Parent | on HN ]

Re:Audio AI: isolating vocals from stereo music using Convol... by lightedman on Saturday February 16, 2019 @12:51AM (#19177299)

Learn how to do waveform inversions - if you have a stereo signal, anything not fully-centered will come through better while the rest is cut out. You can then take that, invert it, and play it back with the original, cutting out that noise and keeping the fully-centered things like vocals present.

This is how I play guitar to my favorite songs on my computer.

[ Parent | on HN ]

Re:Audio AI: isolating vocals from stereo music using Convol... by proctor on Saturday February 16, 2019 @01:16PM (#19179998)

this sounds interesting. could you elaborate a bit? it is unclear if you are inverting once, or twice. "if you have a stereo signal, anything not fully-centered will come through better while the rest is cut out" -- is this before or after an inversion?

[ Parent | on HN ]

Re:Audio AI: isolating vocals from stereo music using Convol... by thibauts on Saturday February 16, 2019 @03:09PM (#19180667)

Of stereo tracks L and R, you invert R and add it to L, effectively canceling anything centered. This usually removes voices. If you subtract (invert then add) the result from the original L and R tracks you get centered sounds only. Results range from perfect to not effective at all depending on the songs you apply it to.

[ Parent | on HN ]

Re:Audio AI: isolating vocals from stereo music using Convol... by lightedman on Saturday February 16, 2019 @10:53PM (#19182583)

This is correct. It is a shame there are no music players that can do this natively. I have to do every track I wish to play along with by hand in Audacity (or if I'm using my Win2K laptop, Cool Edit Pro 2000.)

Alternatively (in theory, anyways,) you can do a spectral analysis, create a selection/range of frequencies to invert, and that can usually handle isolating individual instruments regardless of stereo isolation as long as the other instrument octave ranges do not overlap with the one you're isolating. I'd love a music player that could do that.

[ Parent | on HN ]

Re:Audio AI: isolating vocals from stereo music using Convol... by wodenokoto on Saturday February 16, 2019 @12:12PM (#19179583)

> but it seems it was done without reference to academic literature on source separation

Sums up towards data science pretty well.

[ Parent | on HN ]