StarCoder and StarCoderBase: 15.5B parameter models with 8K context length

Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by csdvrx on Monday May 15, 2023 @06:55PM (#35955625)

Have you tried quantization? It's often a cheap and simple way to reduce the VRAM requirements.

What hardware are you using? (CPU,RAM,GPU,VRAM)

Have you considered using llama.cpp for a mixed CPU+GPU use (if you have enough RAM)

[ Parent | on HN ]

Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by fbodz on Monday May 15, 2023 @07:12PM (#35955796)

Yeah I am using the default training script with int8 quantisation. It uses peft with lora but this still requires 26gb

[ Parent | on HN ]

Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by freeqaz on Monday May 15, 2023 @07:57PM (#35956170)

Is it possible to offload some layers to CPU and still train in a reasonable amount of time?

[ Parent | on HN ]

Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by generalizations on Monday May 15, 2023 @08:15PM (#35956297)

There’s also that pruning tool that was on hn in the last couple weeks. It seemed to work really well on the larger models, and could reduce size by 30-50%

[ Parent | on HN ]

Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by int_19h on Monday May 15, 2023 @09:28PM (#35956761)

I'm not sure about this model specifically, but training with 4-bit quantization has been a thing with LLaMA for a while now, although the setup involves manual hacks of various libraries.

[ Parent | on HN ]

Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by nl on Tuesday May 16, 2023 @02:48AM (#35958456)

You probably don't want to fine-tune a quantized model. They are fine for inference but not great for training.

[ Parent | on HN ]