News for hackers. Stuff that matters.
Re:Ask HN: Is there an ML model that can go from an audio so... by billconan on Wednesday June 07, 2023 @01:46PM (#36230440)
https://basicpitch.spotify.com/
[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by uptownfunk on Wednesday June 07, 2023 @02:02PM (#36230729)
Thanks, I need something that can handle multiple instruments in the recording
[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by ksherlock on Friday June 09, 2023 @12:53PM (#36260615)
spleeter, demucs (my favorite), etc, can split a song into 4 tracks (drum, bass, vocal, and other). Your other comments mentioned vocals, percussion (hand drums and claps), and organ; on paper it sounds like it should work.
[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by esafak on Wednesday June 07, 2023 @02:04PM (#36230761)
wow, how did you find this? I wonder what else they have that they don't show.
[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by n3v3r3v3r on Wednesday June 07, 2023 @02:25PM (#36231155)
<https://github.com/spotify> seems to be the place to look
[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by tmaly on Wednesday June 07, 2023 @01:50PM (#36230520)
Ableton Live has something that can convert audio to midi.

Then there are open source programs that can convert midi to sheet music.

I have tried the process, but you tend to get artifacts in the midi from Ableton.

You have to know a little bit about music theory to clean it up.

[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by uptownfunk on Wednesday June 07, 2023 @02:02PM (#36230723)
Oh cool, do you know if this works for recordings where there are multiple instruments (clapping, organ, vocals, hand drums)
[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by jrflowers on Wednesday June 07, 2023 @02:39PM (#36231395)
I imagine that if you separated a track into its components you could extract the sheet music for each part individually. I don’t know how sheet music for clapping would work though.
[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by solarmist on Wednesday June 07, 2023 @04:10PM (#36232827)
I imagine just notes on the C line with an x plus stem like is used for basic drum rhythm notation.
[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by tmaly on Monday June 12, 2023 @03:57PM (#36300029)
It does, but it does not break them out in to tracks.
[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by Lariscus on Wednesday June 07, 2023 @10:03PM (#36236247)
Is that worth it tough? It seems to me the effort of cleaning up sheet music generated from MIDI would be higher than transcribing the whole piece by hand.
[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by tmaly on Monday June 12, 2023 @03:56PM (#36300022)
It depends on how good you are at music theory.

Some people can do this cleanup really quickly.

[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by ndstephens on Wednesday June 07, 2023 @03:33PM (#36232253)
https://www.celemony.com/en/melodyne/what-is-melodyne

This might help. Not exactly what you're looking for.

[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by ikhatri on Wednesday June 07, 2023 @03:40PM (#36232378)
I was going to post basic pitch from Spotify but it looks like billconan beat me to it. That said I can give you a bit more advice. The Spotify basic pitch model isn't too good at multi-track input. It's capable of it, but you may actually get better results if you separate out the tracks first and then run them individually through the basic pitch model.

In order to do this you can use a source/stem separation model like spleeter (https://github.com/deezer/spleeter) and then run the basic pitch model (or any other midi transcription model). There's other you can try which may yield better results, for example: (https://github.com/Music-and-Culture-Technology-Lab/omnizart)

Either way the key words you want to be looking for are "midi transcription" and "stem separation", should help you find more models to try for both steps. Good luck! :)

[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by crtified on Wednesday June 07, 2023 @06:32PM (#36234583)
That is insane(ly cool). Do you know how the separation models work, in principle?
[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by ikhatri on Wednesday June 07, 2023 @06:49PM (#36234753)
Yup! It's a U-Net style convolutional neural network that runs on the spectrograms. There's a short 2 page abstract here: https://archives.ismir.net/ismir2019/latebreaking/000036.pdf and they cite this prior work as where they got the architecture from: https://arxiv.org/abs/1906.02618
[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by radq on Wednesday June 07, 2023 @09:30PM (#36236035)
I've been working on training models for this problem. Haven't shipped as a product yet because the inference cost is too high (takes 10-15 minutes to transcribe each song). Would love to try it out on the music you're trying to learn and get your feedback. Email address is in my profile.
[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by moomoo11 on Thursday June 08, 2023 @02:39PM (#36246580)
If someone's seriously working on this, I would be happy to help out and build the mobile apps around it.

Imagine how cool, useful, and joyful this tech would be to most people.

[ Parent | on HN ]
Re:Ask HN: Is there an ML model that can go from an audio so... by CrypticShift on Friday June 09, 2023 @02:29PM (#36262028)
This is called (multitrack) music transcription. There are some commercial solutions (AudioScore, AnthemScore, ...). For OSS, look at Omnizart [1] and magenta/mt3 [2].

I suppose these models are trained on western / pop music, so they may not work nicely on ethnic music.

[1] https://github.com/Music-and-Culture-Technology-Lab/omnizart [2] https://github.com/magenta/mt3

[ Parent | on HN ]