Re:PMC-LLaMA: Further Finetuning LLaMA on Medical Papers
by zwaps on Tuesday May 09, 2023 @03:07AM
(#35871081)
Also, if I am not reading this wrong, the paper shows that when Fine-tuning on a task, this additional Medical training (7 days on A100) essentially does nothing (benefit of maybe 1% over standard LLaMA).
Zero shot comparisons are not shown.
This looks like a failed experiment to me?
However, in the paper they say that for each of the 5 epochs, they sample 512 tokens from each paper randomly.
This would come down to an effective training size of 12 billion tokens.