Sure, we could see `animal = f(image)` as a partial function and lift it into some total function space, `maybe_animal = f(image)`.

Is our reasoning here actually this total function though? We really do always need to keep in mind that animal reasoning will terminate, animals will act "guided by other reasoning", and resume the original reasoning process.

Action really messes up this neat "totalising option" for partial functions. If i'm not sure if "fluffy" is a dog or a cat, i might wait longer for it to move; I might throw something at it; I might ask its owner.

This isnt as simple as "don't know", since reasoning is kinda time-bound and time-parameterised, my very measuring process is sensitive to my own confidence in what-something-is.

Is this really "f(image) = dont-know"? I dont think so.

I think it's more like, "judgement = body-brain-state(t, reasoning-goal, {action policies}, ...large-number-of-other-things)".

This gets to my issue. In my view what animals are doing is better described by a dynamical equation of state (like a wavefunction), as the whole system is operating under its own dynamical evolution (including, eg., what fluffy does in response to your puzzlement).

I dont see Dog|Cat|DontKnow as the answer here. I dont think it's the actual total function which corresponds to our judgement, though it probably is total -- we just end up with "DontKnow" in all the cases actual intellkigence is requireds.... the very think we're aiming to model.

"NNs are bad at extrapolating, including e.g. trivially to periodic functions."

WaveNet and its descendents are the obvious counter example here. It is excellent at learning to generate periodic and nearly periodic functions...

I know I've seen papers where they use sinusoidal nonlinearities to learn periodic functions. I felt like that's a bit of a hack though (not necessarily a bad thing)- you're bringing domain knowledge in, which if you allow makes it easy to extend to periodic functions. The failure of a vanilla nn to learn periodicity is I think a specific failure to extrapolate, which is the bigger problem.

I'll take a look at the architecture you mention.

Edit: looked it up, I see it's an auto regressive model, like pixel CNN in 1D. I do know pixelCNN, I hadn't really considered the connection to periodic functions but I see what you're saying. In a sense any AR model is extrapolating, but not in the sense I mean: it has seen training examples of the next point predicted from the last points, it's not extending a relationship it learned in training to something new. Anyway, thanks for pointing the model out

There's ample literature showing that neutral vocoders generalize to new speakers not seen in training. IME, once you train on around ten speakers the models generalize quite well to new speakers, and in practice training sets for production models are often using hundreds of speakers for training.

There are also non autoregressive GAN vocoders which work great, with a tiny fraction of the compute of the original WaveNet. Soundstream is a good example of this.

My own work on bird song separation uses non AR models for generating separation masks. It works great and generalizes to new habitats and species without difficulty. https://ai.googleblog.com/2022/01/separating-birdsong-in-wil...

Calling architecture variants a hack is risking no true Scotsman arguments... The only true neutral networks are the ones that don't work, perhaps?

I think the approach of [0] is closer to what’s going on in our brains, basically evolving symbolic equations of partial derivatives until we get something good enough. Really fascinating and succinct paper.

0. https://cdanfort.w3.uvm.edu/courses/237/schmidt-lipson-2009....