I know I've seen papers where they use sinusoidal nonlinearities to learn periodic functions. I felt like that's a bit of a hack though (not necessarily a bad thing)- you're bringing domain knowledge in, which if you allow makes it easy to extend to periodic functions. The failure of a vanilla nn to learn periodicity is I think a specific failure to extrapolate, which is the bigger problem.

I'll take a look at the architecture you mention.

Edit: looked it up, I see it's an auto regressive model, like pixel CNN in 1D. I do know pixelCNN, I hadn't really considered the connection to periodic functions but I see what you're saying. In a sense any AR model is extrapolating, but not in the sense I mean: it has seen training examples of the next point predicted from the last points, it's not extending a relationship it learned in training to something new. Anyway, thanks for pointing the model out

There's ample literature showing that neutral vocoders generalize to new speakers not seen in training. IME, once you train on around ten speakers the models generalize quite well to new speakers, and in practice training sets for production models are often using hundreds of speakers for training.

There are also non autoregressive GAN vocoders which work great, with a tiny fraction of the compute of the original WaveNet. Soundstream is a good example of this.

My own work on bird song separation uses non AR models for generating separation masks. It works great and generalizes to new habitats and species without difficulty. https://ai.googleblog.com/2022/01/separating-birdsong-in-wil...

Calling architecture variants a hack is risking no true Scotsman arguments... The only true neutral networks are the ones that don't work, perhaps?

I think the approach of [0] is closer to what’s going on in our brains, basically evolving symbolic equations of partial derivatives until we get something good enough. Really fascinating and succinct paper.

0. https://cdanfort.w3.uvm.edu/courses/237/schmidt-lipson-2009....