News for hackers. Stuff that matters.

A comment on Eliezer Yudkowsky on large language model economics. Parent comment

Re:Eliezer Yudkowsky on large language model economics by terafo on Tuesday March 14, 2023 @11:57AM (#35153599)
You don't need to store it in weights. It would be too expensive to constantly update model with new data. It wouldn't be able to contain everything anyways. LLaMa 30B with access to 100TB of storage(you can store whole libgen plus arxiv and much much more in there) and trained to use it should be ahead of LLaMa 65B with no such access in terms of world knowledge. And it would cost less.
[ Parent | on HN ]
Re:Eliezer Yudkowsky on large language model economics by Imnimo on Tuesday March 14, 2023 @12:43PM (#35154337)
You can retrieve small amounts of specific information this way. But suppose the model was asked to write code in a new programming language that it doesn't have existing knowledge of. Could it do that given unguided access to the docs?
[ Parent | on HN ]
Re:Eliezer Yudkowsky on large language model economics by terafo on Tuesday March 14, 2023 @01:15PM (#35154856)
It might, but in this specific instance some kind of fine-tuning would be better.
[ Parent | on HN ]
Re:Eliezer Yudkowsky on large language model economics by Imnimo on Tuesday March 14, 2023 @01:50PM (#35155497)
It feels pretty obvious to me that it could not, but then again I've been wrong about LLM abilities before!
[ Parent | on HN ]
Re:Eliezer Yudkowsky on large language model economics by webmaven on Tuesday March 14, 2023 @06:37PM (#35160117)
We've already seen zero-shot learning of novel syntaxes (that is, chatting with an LLM to interactively devise a new language that is then used, without any actual retraining of the model).
[ Parent | on HN ]