How do you compare them to Aiva [1]?
Aiva is way better than this honestly, and I mean the pure AI part of it, that (AIUI) only really generates monotimbral, piano-like music (the orchestration in the full pieces they release was - by their own admission - done by humans, last I looked into it). You can actually tell that Aiva involves real serendipitous ("out-of-sample") creativity of the sort that AIs (and good human composers!) are best at. (But the openly-available models I mentioned elsewhere in the thread are still a bit better, TBH. At least IMHO, and when it comes to rewarding active listening.)
openai says they can do it:
Long-term structure MuseNet uses the recompute and optimized kernels of Sparse Transformer to train a 72-layer network with 24 attention heads—with full attention over a context of 4096 tokens. This long context may be one reason why it is able to remember long term structure in a piece
the problem to the scaling thing is that we don't nearly have as many samples per artist as we do have "text pointed to by links on reddit", as used in GPT2 (openAi's text generation thing). I would expect that they can't go as crazy as they could
Most previous attempts at neural net composition restricted the training set to one style of music or even one composer, which is pretty silly if you understand how neural nets work. It was obvious to me if you used a very large network, chose the right input representation, and most importantly used a complete dataset of all available music, you would get great results. That's exactly what OpenAI has done here.
It is still lacking some longer term structure in the music it generates (e.g. ABA form). But I think simply scaling up further (model and dataset size both) could fix that without any breakthroughs. This seems to be OpenAI's bread and butter now: taking existing techniques and scaling them up with a few tweaks. (To be clear, I don't mean to minimize what they've done at all. "Simply" scaling up is not so simple in reality.)
What might still need some breakthroughs is applying the same technique to raw audio instead of MIDI. Perhaps what is needed is a more expressive symbolic representation than MIDI. I'm imagining an architecture with three parts: a transcription network to produce a symbolic representation (perhaps embedding vectors instead of MIDI), something like this MuseNet for the middle, and a synthesis network to translate back to raw audio. This would be analogous to gluing together a speech recognizer, text processing network, and a speech synthesizer. Such a system could generate much more natural sounding music, even perhaps with lyrics.