Music is hard. It's far harder than most people realise.
Winning a game is a relatively easy problem, because you know when you have won. Music is much more nebulous. Grammatical correctness is preparatory to creative intent. Even basic metrics of correctness are rather fuzzy.
ML doesn't have any concept of intent, so it tends to flail around in a statistical space - and to sound like it.
Suppose an AI created a Bach-like contrapuntal exercise with a lot of cross-relations (i.e., clashing dissonant sounds). Would scholars judge it to be at the level of Bach because of the handling of these numerous cross-relations? Or would they claim it isn't at sophisticated as Bach because having that many cross-relations isn't stylistically accurate? Based on the historiography of musicology I would guess the latter. Even though there's an unfinished extant Bach fugue in C minor where he's essentially role-playing a cross-relation obsessed AI composer.
The history of theory is even worse. For example, Schenker created theories based on the mastery that Bach, Beethoven, and others displayed in their compositions. But when a piece by Bach didn't fit his theory, he claimed that Bach actually wrote it incorrectly!
I'm afraid AI music research is going to have to put up with the same irrationality of humans as every other subjective endeavor. That is, musicians will kick and scream that AI "isn't doing it right," then when it inevitably becomes embedded as a core part of music making we'll hear all about how music produced with classical algorithms on old protools rigs sounds so much "warmer."
I am only an amateur player of Bach, but I felt the same way. However, it probably is worth noting that a major reason why Bach works in this context is the particular nature of his music--or more precisely the particular subsample of his music that GEB refers to. I don't think the music of most other composers, or even some of Bach's less contrapuntal music, would have made sense in this context.
Weaving Bach in makes perfect sense.
Recently, one of the top search based engines (stockfish) added neural networks on top of it, and it got significantly better still. This was a couple months back, maybe the pure neural nets have caught up again!
[Deep Blue] cannot answer elementary questions about chess such as “If a bishop is now on a black square, will it ever be on a white square?” ... All that it can do is to generate a next move of extraordinary quality.
But in a deeper sense, the match between Deep Blue and Kasparov proves that Hofstadter was profoundly right. ... It is precisely this power of abstracting, of stepping outside the game to a higher-level view, that enabled Kasparov to find where Deep Blue was weak and to exploit this to win his match.
Also, I strongly agree that the music/Bach connection in the book makes little sense.
The only aspect of the book that really does not work for me is the attempt to integrate music.
And there are some very interesting insights with benefit of 20 years of hindsight.
Today, open-source chess programs are better than the best humans. (they have been "trained" via a form of adversarial learning.)
And:
Part of this, of course, is nostalgia for the halcyon years of AI when the funding agencies had more money to throw around than there were researchers to take it, and the universities had more tenure-track slots than there were Ph.D’s to fill them;
Biden just announced a desire to DOUBLE American investment in R&D. Such an optimistic time we are at right now.