News for hackers. Stuff that matters.
Re:PMC-LLaMA: Further Finetuning LLaMA on Medical Papers by zwaps on Tuesday May 09, 2023 @02:51AM (#35870979)
It said to be trained on 75 Billion tokens with 8 A100 in 7 days.

However, in the paper they say that for each of the 5 epochs, they sample 512 tokens from each paper randomly.

This would come down to an effective training size of 12 billion tokens.

Re:PMC-LLaMA: Further Finetuning LLaMA on Medical Papers by zwaps on Tuesday May 09, 2023 @03:07AM (#35871081)
Also, if I am not reading this wrong, the paper shows that when Fine-tuning on a task, this additional Medical training (7 days on A100) essentially does nothing (benefit of maybe 1% over standard LLaMA).

Zero shot comparisons are not shown.

This looks like a failed experiment to me?