CodeAlpaca – Instruction following code generation model

Re:CodeAlpaca – Instruction following code generation model by MacsHeadroom on Saturday March 25, 2023 @01:30PM (#35304922)

Far far less. Alpaca-7B's compute cost was around $60-$70 for Stanford and around $0.60 (yes 60 cents) for equivalent fine tunes using the Parameter Efficient Fine Tuning (PEFT) strategy of Low Rank Adapters (LoRA).

The repo above can be replicated for similar costs. Easily less than $10 for up to 30B using LoRA (which requires only 24GB of VRAM for 30B/33B and smaller).

[ Parent | on HN ]

Re:CodeAlpaca – Instruction following code generation model by Art9681 on Saturday March 25, 2023 @08:45PM (#35309311)

I am interested in this. What would be the cost for the best model possible by the public?

[ Parent | on HN ]

Re:CodeAlpaca – Instruction following code generation model by NhanH on Sunday March 26, 2023 @01:38AM (#35311375)

I thought so too, but for newcomers, they should expect to train model a dozen times or so :-)

[ Parent | on HN ]