Re:We need to tell people ChatGPT will lie to them, not deba...
by whimsicalism on Sunday April 09, 2023 @01:34PM
(#35504751)
You are largely wrong, to be honest.
Mostly, you can simply stack more transformers and (as long as you have layer norm), it can train on more data (of fixed sequence length).
There are some tricks & tips, but the vast majority lie outside of the architecture improvements.
Re:We need to tell people ChatGPT will lie to them, not deba...
by f6v on Tuesday April 11, 2023 @10:22AM
(#35525576)
I’m not an expert, but I assume there’s a limit where «simply adding stuff” stops to provide improvements when you take training time into account. And my gut feeling is that true reasoning requires different architectures.
I haven’t said that. I pointed that you can’t simply put more data into it and expect improvements. There’s definitely a lot of engineering done on the models to make them capable of using all the data.
I can give you an example. You can’t simply add more convolutional layers to a CNN and feed more images. The architectures of image models include many tricks, like inception layers.
I suspect there’re similar challenges in LLMs. And I’m not sure whether current ChatGPT can use more data to become better at reasoning, or at some point they’d need to come up with different architectures.