News for hackers. Stuff that matters.

A comment on LoRA: Low-Rank Adaptation of Large Language Models.

Re:LoRA: Low-Rank Adaptation of Large Language Models by michaelhartm on Friday March 24, 2023 @03:15PM (#35294087)
Btw, it's kinda crazy how bad the GPT4-J results in the blog are compared to the Dolly one, which seem pretty good. Do we know why it works so well to use this 50k dataset?
[ Parent | on HN ]
Re:LoRA: Low-Rank Adaptation of Large Language Models by quadrature on Friday March 24, 2023 @04:42PM (#35295156)
Dolly is instruction fine tuned whereas GPT4-J is not. Which means that it doesn't even understand that it is being instructed to do something, it is just doing an autocomplete.
[ Parent | on HN ]