One downside with Jukebox: it takes ~9 hours to generate ~90 seconds of audio (even on a NVIDIA V100 GPU) since it's an auto-regressive model making experimentation and 'co-creation' much harder

>> it takes ~9 hours to generate ~90 seconds of audio (even on a NVIDIA V100 GPU)

btw this is true if you upsample all 3 levels. I have found in practise that you are fine upsampling 2 levels and then using a DAW to "clean up"/"remaster" [0]... again will work for certain sounds better than others.

The last upsampling step is by far the most expensive. so cutting that down cuts total time by 75%

[0] simple hacks such as using a low-pass filter

agreed. full waveform "music" (unlike say MIDI) is just many more variables than a "picture" or even "video". Also you have to "stitch together" a lot of these samples to get anything that resembles a "new" song. More akin to mining... but still its kind of crazy what can be done with it.

>> Why doesn't the architecture use a model to generate sheet music / MIDI, then layer another model on top to create instrumentals?

there are various models that do this [0]... just doesn't have the same power to generate waveform music that jukebox does.

Jukebox effectively is a 2 step model a) a 3 layer VQ-VAE compresses the music (so think of this as the MIDI equivalent) and b) a transfomer then learns/generates sequences.

The compression is the expensive part of the model.

[0] https://towardsdatascience.com/how-to-generate-music-using-a...

I'm not familiar with Jukebox, so maybe I'm wrong. It seems like you'd want a fast architecture to use a model to generate sheet music / MIDI, then layer another model on top to create instrumentals?