Re:80% faster, 50% less memory, 0% loss of accuracy Llama fi...
by danielhanchen on Saturday December 02, 2023 @04:12AM
(#38497250)
You can refer to QLoRA's official finetuning notebook https://colab.research.google.com/drive/1VoYNfYDKcKRQRor98Zb... for your reference!! Obviously I can't provide the code we have, but if you use the same datasets and the same settings (bsz = 2, ga = 4, max_grad_norm = 0.3, num_epochs = 1, seed = 3407, max_seq_len = 2048) you should be able to replicate it.
[ Parent
| on HN ]
There are no comments on this comment yet.