> ‚make the map the size of the territory'[0], which is why they only succeed at a narrow class of interpolation problems

I take it you have not seen the recent Dall-E 2 results? Clearly that model is not just working on a narrow space.

See https://openai.com/dall-e-2/ and the many awe-inspiring examples on Twitter