News for hackers. Stuff that matters.

A comment on Google trained Bard on ChatGPT chats from ShareGPT. Parent comment

Re:Google trained Bard on ChatGPT chats from ShareGPT by ithkuil on Thursday March 30, 2023 @04:10PM (#35378216)
That assumes that training on the output of another language model somehow gives you the ability to improve your model and to catch up somehow
[ Parent | on HN ]
Re:Google trained Bard on ChatGPT chats from ShareGPT by iandanforth on Thursday March 30, 2023 @05:33PM (#35379090)
It does. In general this is known as teacher-student training or knowledge distillation. It works better if you have access to the activations of the model but you can work with just outputs as well.
[ Parent | on HN ]
Re:Google trained Bard on ChatGPT chats from ShareGPT by ithkuil on Friday March 31, 2023 @01:36AM (#35383043)
Does it work even if you don't control the inputs?
[ Parent | on HN ]
Re:Google trained Bard on ChatGPT chats from ShareGPT by satvikpendem on Thursday March 30, 2023 @06:06PM (#35379426)
Well, it does, that's how we got Alpaca from LLaMA.
[ Parent | on HN ]