Simply put, there is less information in music. Copyright infringement cases have been won on as little as a handful of notes. So the likelihood of a copyrightable element from the underlying showing up in the model's output is that much higher. A single melody represents a similar portion of all possible melodies as does a color all possible colors and certainly less than the fraction represented by a single color palette. Yet you cannot copyright a color palette but you can a melody.

Copyright law regarding audio media is entire different beast than visual media; the case law and fair use technicalities do not always line up as simply as "if its legal on visual media it must also be legal on audio."