News for hackers. Stuff that matters.

A comment on Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa. Parent comment

Re:Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generat... by m3kw9 on Tuesday March 28, 2023 @08:35PM (#35350315)
How does training on just 800k pieces of data need 7b parameters?
[ Parent | on HN ]
Re:Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generat... by comex on Tuesday March 28, 2023 @08:40PM (#35350372)
Because it’s fine-tuning an existing 7B-parameter language model, not training from scratch.
[ Parent | on HN ]
Re:Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generat... by sp332 on Tuesday March 28, 2023 @09:06PM (#35350619)
Llama 7B was trained on a trillion tokens. The Lora is a small fraction of extra neurons that get integrated into the structure, and those are what get trained on the new data. It's like fine-tuning but takes less RAM and compute than retraining the whole model.
[ Parent | on HN ]
Re:Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generat... by KRAKRISMOTT on Tuesday March 28, 2023 @08:41PM (#35350394)
The chinchilla formula demands a 20:1 ratio
[ Parent | on HN ]
Re:Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generat... by airstrike on Tuesday March 28, 2023 @08:58PM (#35350550)
"demands" is great
[ Parent | on HN ]
Re:Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generat... by KRAKRISMOTT on Tuesday March 28, 2023 @09:18PM (#35350727)
Hey I didn't invent Chinchilla, blame google for that.
[ Parent | on HN ]
Re:Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generat... by adt on Wednesday March 29, 2023 @03:10AM (#35353697)
DeepMind

(apparently there's some friction between those two!)

[ Parent | on HN ]