I would agree that the manner of reasoning must differ, since the sequence follows bpe tokens rather than logical steps, however who is to say that another form of mathematical reasoning could not lead to valid results?
For instance, Gpt might solve the problem at each output step insofar as required to generate that token.
It certainly iterates over each output token, and the encoding of the problem is equivalent so iterating over the characters of the math problem, roughly speaking. But yes, the iterated output does not follow a logical graph externally, it is token by token. But internally, the network can absolutely follow a more complex graph.
Could you say what you mean by single network path when we speak about attention based architectures?
What sort of operation or information is missing in such architecture?
I am aware of of some results relating to certain graphs, but I do not think this would apply to a text describing a math problem, say.
GPT does not have such a looping/recursive process inside its neural net. It's a fixed depth non-recursive neural net.
You can get it emulate recursive processes by prompting it with tricks like "think step by step". If you describe the addition algorithm you learn in elementary school (e.g. the digit-by-digit, carry if the sum exceeds 9, etc) in sufficient detail, it can execute that algorithm.