Summarized in another comment but I am a bit worried about hand tracking for a couple reasons. Also mentioned in reply to another comment, but will summarize here for convenience as well!
1. While I know there is promising tech in this area (and I am very interested in it!) the tolerance for error while playing/performing music is slim to none. Will a solution end up being 75% accurate? 90%? Both are two low IMO. Pure MIDI keeps things in a relatively more controllable realm
2. Camera rigging. There are just so many variables here for things to go wrong/right. Quality of the camera, position, lighting. It’s challenging just finding a good method of positioning a camera even if you have the right equipment to locate it.
All that said, I’ll still reinvestigate this zone of ideas in the future and your insight is much appreciated! First, need to prove that I can make the simple case work as well as I think I can :)
Here is an example of raw audio to midi: https://magenta.tensorflow.org/onsets-frames
Regardless of the feasibility, it won't be as reliable as direct midi input, that's for sure! I think your goal of getting things working for a simpler case first is right, and wish you best luck :)
It's so helpful to just get into the flow of playing, then the things that teachers tell you make sense.
Most lessons we cover concepts that don’t immediately click until I next sit down on my own and just focus on playing. After getting past the familiarity building, and into the flow state, all the things my teacher said start to make more sense haha. Some of the time at least, but that’s what weekly checkins are for :)
It’s a slow process learning to read music in my thirties, but it’s just as possible as learning anything else, don’t listen to people who say you can’t learn music as an adult :)
Initially I do plan to keep the scope small to see how well it performs, identify the problem areas, etc.