Re:MonadGPT – What would have happened if ChatGPT was invent...
by LightMachine on Saturday November 25, 2023 @11:25PM
(#38419154)
Why don't you restrict the training just to old text? That would be awesome
[ Parent
| on HN ]
Re:MonadGPT – What would have happened if ChatGPT was invent...
by isoprophlex on Saturday November 25, 2023 @11:44PM
(#38419226)
I'm not sure you'd be able to scrape together enough data for that to be viable...
Re:MonadGPT – What would have happened if ChatGPT was invent...
by iopq on Sunday November 26, 2023 @01:08AM
(#38419556)
https://en.wikipedia.org/wiki/Corpus_Inscriptionum_Latinarum
approximately 180,000 inscriptions
Re:MonadGPT – What would have happened if ChatGPT was invent...
by xcv123 on Sunday November 26, 2023 @04:14AM
(#38420259)
That only contains a few million tokens. Useless for pre-training an LLM from scratch. You would need to find billions of tokens.
Re:MonadGPT – What would have happened if ChatGPT was invent...
by iopq on Sunday November 26, 2023 @08:29AM
(#38421434)
It should be similar to this 1700s English model, probably trained on modern data to start and then at the end fitted to the smaller data set
Re:MonadGPT – What would have happened if ChatGPT was invent...
by xcv123 on Sunday November 26, 2023 @01:24PM
(#38423702)
Yes it requires an extremely large diverse training set for the first unsupervised stage (pre-training). Then fine tune it on the smaller data set. But we may need to wait for the next generation of LLMs that incorporate planning algorithms so that it can better stay focused on its goal for whatever tasks we are asking it to do for research purposes. Otherwise we end up with this https://news.ycombinator.com/item?id=38418974
Re:MonadGPT – What would have happened if ChatGPT was invent...
by Dorialexander on Sunday November 26, 2023 @05:38AM
(#38420625)
Not feasible to go with pretraining only.
What is possible is to use a larger learning rate but this will be a hard trade-off with conversational capacities. Fine tuning is currently based on original texts with a synthetic prompt. The issues that people have noticed (repetitions, not remembering what was in the prompt) will be more significant if the learning rate is higher.
Maybe a solution will be to provide two different variant of the same model, one less immersive and more workable, and the other more immersive and buggy.