As you start to get to the size and comprehensiveness in the corpus that’s going on now, novel model outputs approach being something that might not have been in the training set, but likely will be in the future.

Copilot spitting out a function out of its training data with changed variable names to match those in your file(s) - and no one is actually testing what proportions of results those are - is still regurgitation.

The idea that models could possibly overfit the training data is hardly a new idea. It's standard practice to test for that. Check section 7 of the PaLM paper for example. https://arxiv.org/abs/2204.02311

The N in the NLP training set and arch implies a fuzzy match.

Why anyone would rather start with a plausible but broken-in-a-subtle way buffer rather than a blank one is beyond me.