News for hackers. Stuff that matters.

A comment on Falcon LLM – A 40B Model. Parent comment

Re:Falcon LLM – A 40B Model by kbrkbr on Sunday June 18, 2023 @04:11AM (#36378251)
I tried it using oobabooga's webui side by side with Alpaca 65B loaded in 4 bit on the same AWS instance with 64GB of VRAM.

While Alpaca produced 3 tokens/sec, Falcon produced 0.17 tokens/sec.

So it is very slow with the current tooling still.

[ Parent | on HN ]
Re:Falcon LLM – A 40B Model by brianjking on Sunday June 18, 2023 @09:49PM (#36386426)
How can you deploy Oogabooga to AWS/Huggingface/etc?

Any tips?

Cheers!

[ Parent | on HN ]