News for hackers. Stuff that matters.

A comment on ChatGPT's 1-Year Anniversary: Are Open-Source Large Language Models Catching Up?. Parent comment

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by Tostino on Thursday November 30, 2023 @10:33PM (#38482800)
Sure thing, I used axolotl, and my training parameters were:

sequence_len: 6144 lora_r: 128 lora_alpha: 48 learning_rate: 0.00006 warmup_steps: 600 lr_scheduler: cosine gradient_accumulation_steps: 4 micro_batch_size: 1 num_epochs: 4 optimizer: paged_adamw_32bit flash_attention: true sample_packing: true

[ Parent | on HN ]
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by kaycebasques on Friday December 01, 2023 @10:46AM (#38488101)
Thanks for sharing all these details throughout the thread, Tostino. True open source spirit.
[ Parent | on HN ]