> It might be possible to do hand tracking like that with an iphone
Yeah I’ve been floating ideas like that in my head as well, in fact another commenter just pointed out some promising tech. I’m still on the fence about the value add compared to cost of implementation and accuracy. One reason I like sticking to MIDI only is that it’s much much easier to keep it consistently good (assuming underlying connection logic is sound). The margin for error is so low here, you just can’t have the tech getting in the way of the music is what I am putting my bets on. I’m worried that hand tracking will end up being months of effort for something that ends up being about 75% accurate. Number is completely made up but hopefully my point comes across :)
> Have you tried concerting sound from a phone microphone to midi?
If you mean converting audio data to midi, I’ve thought about it and ran away quickly after haha. Whoever solves this problem to a high enough degree of accuracy to be useful will be sitting on something quite valuable. I have my goal posts set a bit lower at first, on things I am confident I can deliver to the public, but I plan to revisit this in the future :)
Initially I do plan to keep the scope small to see how well it performs, identify the problem areas, etc.
Summarized in another comment but I am a bit worried about hand tracking for a couple reasons. Also mentioned in reply to another comment, but will summarize here for convenience as well!
1. While I know there is promising tech in this area (and I am very interested in it!) the tolerance for error while playing/performing music is slim to none. Will a solution end up being 75% accurate? 90%? Both are two low IMO. Pure MIDI keeps things in a relatively more controllable realm
2. Camera rigging. There are just so many variables here for things to go wrong/right. Quality of the camera, position, lighting. It’s challenging just finding a good method of positioning a camera even if you have the right equipment to locate it.
All that said, I’ll still reinvestigate this zone of ideas in the future and your insight is much appreciated! First, need to prove that I can make the simple case work as well as I think I can :)
Here is an example of raw audio to midi: https://magenta.tensorflow.org/onsets-frames
Regardless of the feasibility, it won't be as reliable as direct midi input, that's for sure! I think your goal of getting things working for a simpler case first is right, and wish you best luck :)
It's so helpful to just get into the flow of playing, then the things that teachers tell you make sense.
Most lessons we cover concepts that don’t immediately click until I next sit down on my own and just focus on playing. After getting past the familiarity building, and into the flow state, all the things my teacher said start to make more sense haha. Some of the time at least, but that’s what weekly checkins are for :)
It’s a slow process learning to read music in my thirties, but it’s just as possible as learning anything else, don’t listen to people who say you can’t learn music as an adult :)
The keyboard highlights engaged keys with color already, so I have that to play with as well as the speed and fully-depressed angle of the key itself.
VMPK is a big influence, which respects note velocity. https://vmpk.sourceforge.io/
Project name is Midishare.
Got the idea after starting piano lessons about 5 months ago. It’s all over zoom, which works surprisingly well for music lessons on its own, but it’s difficult rigging a camera to show the remote person what you’re doing on the keyboard, as well as getting the sound to come through (if you don’t have a nice audio interface). There is still the issue of communicating finger position, not quite sure how to solve that one yet, or if it really even warrants a technical solution (again, you also have zoom to just communicate verbally, works okay for fingering)
The keyboard I use to animate playback is a 3D model which communicates the flow of playback surprisingly well, it’s at least a pretty cool accomplishment on its own!
I’ll launch it with a Show HN one of these days, within a month or two is the goal!