StarCoder and StarCoderBase: 15.5B parameter models with 8K context length

Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by jimlongton on Monday May 15, 2023 @06:19PM (#35955272)

(Possibly naive question) This is marketed as open source. Does that mean I can download the model and run it locally? If so, what kind of GPU would I need?

[ Parent | on HN ]

Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by pyrophane on Monday May 15, 2023 @06:32PM (#35955410)

Here is a good reference:

https://huggingface.co/docs/transformers/perf_train_gpu_one

[ Parent | on HN ]

Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by joaogante on Wednesday May 17, 2023 @01:01PM (#35978155)

A 3090 (or any GPU with >=20GB VRAM) can run StarCoder with int8 quantization at about 12 tokens per second, 33 with assisted generation -- which will come out for StarCoder in the coming days.

When 4-bit quantization comes out, I would expect a GPU with 12GB VRAM to be able to run it.

Disclaimer: I work at Hugging Face

[ Parent | on HN ]