Hello Dolly: Democratizing the magic of ChatGPT with open models

Re:Hello Dolly: Democratizing the magic of ChatGPT with open... by aabajian on Friday March 24, 2023 @02:14PM (#35293304)

Anyone care to comment on why the output of these models changes so dramatically given so little Q&A training? It's a 6 billion parameter model with only 50 thousand Q&A samples.

It's clear the model already "knows" the format of a Tweet (short length, attention-grabbing, contains hashtags). The model also knows stuff about language models (word2vec, tokenization), and can include entities from the question in its response (Dolly, Databricks). Yet, it just doesn't put these pieces together in the right way without the Q&A training.

Edit: For kicks, I asked GPT-4 this question: https://imgur.com/a/sM4uyBn

[ Parent | on HN ]

Re:Hello Dolly: Democratizing the magic of ChatGPT with open... by pwendell on Friday March 24, 2023 @04:02PM (#35294663)

Yes this was a very surprising result... that the relatively small uptraining was able to unlock so much latent knowledge in the model.

[ Parent | on HN ]

Re:Hello Dolly: Democratizing the magic of ChatGPT with open... by Davexon on Wednesday March 29, 2023 @01:42PM (#35360606)

Just like in Alpaca, the answer lies in the LLama dataset where the focus is on instruction following. GPT-J knows all the answers but just now was taught to understand the questions thoroughly before writing the answer.

[ Parent | on HN ]