Midi is 'lossy' in the sense that it flattens a bunch of knowledge about how a piece is to be rendered into note-on/note-off pairs. You then have to do all kinds of tricks to even begin to approach what the score looked originally. Lots of enharmonically (identically sounding) combinations may have looked completely different on the original score, and when you start adding the various chord combinations it gets even more complicated. Even so, I'm strongly convinced it can be solved and we have made considerable progress on this already.
The 'wiggle room' aspect of it has been addressed to some extent. For instance, we build up a map of all the notes that are sounding at a given time, and will ignore notes that are technically 'wrong' but that are still sounding. This allows for much smoother restarts after mistakes.
Your final point is a bit tricky, but thank you for the hint, I will definitely add it to the list of things to look at to see if we can find a way to make that work.
For my final point, you can model this as a hidden markov process. You're observing notes, but not the user moving his mental "cursor". Classical filtering algorithms should do well here.
Ah, one more missing feature, good scores have fingering, but it gets lost in the midi. I'm pretty sure fingering can be infered though and this would be pretty powerful. Amusingly, it can also be seen as a filtering problem:
- At any given point, a hand can have a finite number of fingerings, and that number isn't that large. Give each of them a difficulty.
- Now, make that difficulty conditional on the previous hand position.
- Run Viterbi's algorithm on the score to minimize the difficulty.
The difficulty can be obtained from an expert or through some physiological model, but they could also be inferred from an existing body of music sheets.
Importing alternative formats (xml for instance) is definitely an option worth looking into, but because the bulk of the material that is out there that people can freely access is in midi format we chose that one first.
Musescore already has such a collection of sheet music https://musescore.com/hub/piano so it'd also be possible to allow users to import from there, going from sheet music to midi instead of trying to reverse the process.
As using MIDI will probably never be perfect, have you thought about using the score in a suitable format, like LilyPond, directly?
A midi recording contains pretty precise velocity info for each note (0-127) that is either not present at all in a score, or is present only in the form of a limited number of dynamic markings[1], which govern the flow of dynamics imprecisely, and almost not at all on a per-note basis.
With tempo it is similar to dynamics: midi will precisely determine when notes turn on and off, down to the millisecond, in the process resulting in the midi performance having a precise tempo. In contrast, musical scores have limited means to indicate proper tempo[2], necessitating (as with dynamic markings in a musical score) much interpretation by the musician.
All of this is not to say that a midi performance is somehow "better" than a score. There are good midi recordings and bad ones. A good recording will generally be made by an accomplished musician who in making the recording records fairly precisely a particular (and good) interpretation of the score. Likewise, a bad recording would generally be the result of a poor interpretation of a score. But whatever the case, the midi recording is far more precise than the score itself.
[1] https://www.aboutmusictheory.com/music-dynamics.html
[2] https://en.wikipedia.org/wiki/Tempo#Basic_tempo_markings
A midi file is designed to contain different kinds of information than a score, so it can absolutely contain less information (of a certain kind) than a score.
Imagine a group of notes that are attacked very close together in time in a midi file - is it meant to be a chord played without arpeggiation? A chord arpeggiated as fast as possible? A chord arpeggiated to a specific tempo?
When a composer writes (say) a chord with an arpeggiation sign next to it, it specifies that the notes are to be played in a certain order, and quickly. There are multiple possible performances that comply with this "specification". MIDI cannot encode this information (the information being the set of acceptable performances).
The way I learn is by bars, repeatedly re-playing bars say, 1 to 4, until I get them, them 3 to 7, until I get them, etc.. But most piano apps want you to start playing from start to end slow, then faster, etc.
So: piano apps should allow breaking a piece into bars, allow you to repeat those bars again and again until you exceed certain score. Maybe even keep track of which bars are harder to play, and repeating the difficult bars more often, etc.
(Not affiliated, just something I personally use)
I'm a piano noob but wouldn't you want to practice catching back up with the beat if you ever intend to perform?
I'm not quite happy with Synthesia. There is no visual feedback on completing a note. This has been a staple feature in Doremi Mania and Keyboard Mania for twenty years, so they have zero excuse for not implementing it.
On the off-chance that this will inspire the article's author for their development, or simply for the curious, this has been my experience with it:
The primary mode of display is "Guitar Hero" style (it's hard to describe just look at this video to get an idea[2]) but it's possible to display the sheet music. It lets you practice and grades you depending on how close you're sticking to the score. It's very nice to be able to practice one hand separately while having the app play the other hand. I do fairly well reading sheet music but the first time deciphering it can be a bit tedious, using the app speeds this up considerably for me.
Synthesia is almost great but it has a lot of small shortcomings which add up to a bad user experience, for example:
- The conversion from MIDI to a readable score is very poor compared to, say, Musescore. The sharps and flats are generally off (e.g. randomly a double flat D will appear instead of a C, even though the score has a C, Musescore interprets the midi as a C, etc).
- Not enough wiggle room when playing, e.g. if a chord is not arpeggiated in the midi file, you won't be able to move forward by arpeggiating.
- The app really depends on you following the score exactly and linearly. It should also let the score follow you. For example, if you're playing at a given pace, miss, stop and repeat a bar, the app should detect that automatically instead of just carrying on expecting you to keep playing.
[1] https://synthesiagame.com/
[2] https://www.youtube.com/watch?v=sleZ-hzrtRY