Re:Transformers struggle with generalizing tasks beyond pre-...
by spywaregorilla on Tuesday November 07, 2023 @04:41PM
(#38183272)
Because the optimization logic of tweaking the gpt transforms via training data and nudging weights and biases to achieve token prediction is not a path to utilizing the full potential of the gpt transforms.
[ Parent
| on HN ]
Re:Transformers struggle with generalizing tasks beyond pre-...
by criddell on Tuesday November 07, 2023 @05:05PM
(#38183620)
I don’t see why not.
I haven’t tried this, but I think I could describe a very basic logical CPU to it and have it execute opcodes and tell me the CPU state at each step (like Knuth’s MIX). Couldn’t I ask it to write programs and execute programs for this computer when doing so would help it give better answers?
Re:Transformers struggle with generalizing tasks beyond pre-...
by spywaregorilla on Tuesday November 07, 2023 @08:00PM
(#38185398)
Highly unlikely to work in the near future. LLMs are bad state machines.
But even if it did, you are basically describing programming in a contrived language, not the llm learning to do math. You can explicitly teach a model to follow an algorithm. But that's not the same as the algorithm being "learned" during training and "invoked" when asked what's 2 + 2?