Q: If I could give you a black box that could do anything, what would you have it do?
A: I would love to have a box onto which I could offload choice making. A thing that makes choices about its outputs, and says to itself, This is a good output, reinforce that, or replay it, or feed it back in. I would love to have this machine stand for me. I could program this box to be my particular taste and interest in things.
Q: Why do you want to do that? You have you.
A: Yes, I have me. But I want to be able to sell systems for making my music as well as selling pieces of music. In the future, you won't buy artists' works; you'll buy software that makes original pieces of "their" works, or that recreates their way of looking at things. You could buy a Shostakovich box, or you could buy a Brahms box. You might want some Shostakovich slow-movement-like music to be generated. So then you use that box. Or you could buy a Brian Eno box. So then I would need to put in this box a device that represents my taste for choosing pieces.
Enough of these passes and detailed removal of elements you wouldn't create, and you should get closer and closer to a machine that would be your musical clone.
I'm sure a musician who was also a programmer who understood their own tastes and creation process well enough could currently create something like this for a specific genre, but I think we're very far from a one-size-fits-all generator.
I remember a few years back there was some quick guide to visual design that I saw that recommended this. To provide what appeared to be randomness over repeating samples (or at least prevent easy pattern matching in the brain which is distracting), take three images that can of different prime number lengths, then repeat each one and put them next to each other.
I believe the example used was the ruffles in a stage curtain, where there were a few layers. Each layer was a repeating image of length 3, then length 5, then length 7 (but the image itself had some variations between ruffles within it). You won't get a point where all the images all stop and start at the same point (a dead giveaway of the pattern) until length 105.
Eno's "Music for Airports" famously uses a system of multiple tape loops that produce sequences of different periods. As you listen, you can hear phrases that occur nearly together and then later, well-separated in time, as the periods of these loops go in and out of phase:
"One of the notes repeats every 23 1/2 seconds. It is in fact a long loop running around a series of tubular aluminum chairs in Conny Plank's studio. The next lowest loop repeats every 25 7/8 seconds or something like that. The third one every 29 15/16 seconds or something. What I mean is they all repeat in cycles that are called incommensurable — they are not likely to come back into sync again."
This interaction of periods will have long memory. (If the tape lengths are L and L+d, for small d, then the repeat time could easily be as long as L*(L/d), and even longer if d does not divide L evenly.) Thus, it is very different from what you get with a Markov chain.