Not sure I track. It would help to know where you're coming from.

Given a long enough life-span, a lot of pencil and paper, and some dice, I could do the forward passes of GPT and "write novel code", without there having been any reasoning about the code I'm writing down - I wouldn't even need to know what the code is about.

I mean in theory I could sit and calculate electrical and chemical potentials and interactions and figure out what your next comment will be.

Hard disagree ;)

if we knew you were religious, which we would do if we had a mapping of your brain, we would have known you were going to make this comment