Re:The tide is shifting: 1.3B outperforms 7B Llama 2
by YetAnotherNick on Tuesday September 12, 2023 @02:56PM
(#37486034)
Llama generally acheives much higher accuracy with very small amount of fine-tuning(on similar quality of dataset like this paper) on lot of tasks. So the model understanding is present in llama to get higher accuracy. e.g Hellaswag, ARC and MMLU for 7b model is 0.8, 0.57 and 0.52 respectively[0], while phi-1 is 0.48, 0.45 and 0.38.
I don't think finetuning phi-1 on good quality synthetic data will increase its accuracy as it is only trained on that.
I don't think finetuning phi-1 on good quality synthetic data will increase its accuracy as it is only trained on that.
[0]: https://huggingface.co/pankajmathur/orca_mini_v3_7b