News for hackers. Stuff that matters.
Re:My approach to automatic musical composition by p1esk on Sunday February 13, 2022 @06:58PM (#30326906)
A transformer architecture does not change whether you use word tokens, musical chord encodings, audio waveform samples, image patches, or video frames. You can change a number of layers or attention heads, but the architecture remains the same - it does not care what is the nature of sequences it encodes/decodes. That's kind of the point - to have a universal algorithm. Human neocortex does not care which sensory receptors the signals are coming from, there's evidence that it processes all of them in the same manner.

Any prior information about the domain can be encoded in a way you prepare the input samples, and this would be equally useful and relevant regardless of the DL architecture you decide to use (RNNs, autoencoders, GANs, transformers).

Using raw audio samples has two advantages: First, you provide all available training data to a model, and trust that it will learn to extract what it needs from it. Second, unlike any other music representation formats, you have huge amounts of training data. So essentially all you need is a huge transformer and many thousands of GPUs to train it on, using all mp3s you can get your hands on, and you can reasonably expect a similar level of quality as the quality of text generated by GPT-3. What do you think?

Re:My approach to automatic musical composition by zozbot234 on Sunday February 13, 2022 @07:25PM (#30327096)
> Any prior information about the domain can be encoded in a way you prepare the input samples

Adding features to provide further information about the input data can be helpful, but not nearly as much as ensuring that strong priors are reflected in the actual model architecture, whenever relevant to the domain. Johnson's paper about biaxial RNN provides one example of how to do this in the context of music.