Re:Learnings from employing ChatGPT as a ML Engineer for a day
by visarga on Wednesday February 22, 2023 @03:49PM
(#34902491)
It doesn't memorize training data except for very repetitive bits. It needs a fresh copy of the paper in the prompt but only 10% fits.
[ Parent
| on HN ]
Re:Learnings from employing ChatGPT as a ML Engineer for a day
by sebzim4500 on Wednesday February 22, 2023 @05:10PM
(#34903674)
Making the context 10x longer would 'only' cost about 100x as much compute. Presumably you could take GPT-3 (trained with the normal context) and then finetune it on a comparitively small amount of data with the new context length, so it shouldn't be enormously expensive to train either.
Would be interesting to see how well it could 'understand' a paper, or if more layers etc. would be needed.