I've seen some papers about optimizing piano fingering on arxiv before. They used various kinds of machine learning. Some papers dealing with guitar fingering, too.