News for hackers. Stuff that matters.
Re:Learnings from employing ChatGPT as a ML Engineer for a day by visarga on Wednesday February 22, 2023 @03:49PM (#34902491)
It doesn't memorize training data except for very repetitive bits. It needs a fresh copy of the paper in the prompt but only 10% fits.
Re:Learnings from employing ChatGPT as a ML Engineer for a day by sebzim4500 on Wednesday February 22, 2023 @05:10PM (#34903674)
Making the context 10x longer would 'only' cost about 100x as much compute. Presumably you could take GPT-3 (trained with the normal context) and then finetune it on a comparitively small amount of data with the new context length, so it shouldn't be enormously expensive to train either.

Would be interesting to see how well it could 'understand' a paper, or if more layers etc. would be needed.

Re:Learnings from employing ChatGPT as a ML Engineer for a day by visarga on Thursday February 23, 2023 @12:11AM (#34907019)
Well, it depends. Perceiver can handle big context.

https://arxiv.org/abs/2103.03206