News for hackers. Stuff that matters.

A comment on Open source ChatGPT with LLaMA implementation trains 15x faster. Parent comment

Re:Open source ChatGPT with LLaMA implementation trains 15x ... by pumanoir on Monday February 27, 2023 @12:58PM (#34959696)
I think is feasible. The description even says is designed to save on vram[1]. I don't get the other comments about needing more vram than a 3090.

Also, Neuralmagic may run their sparsification on ARM cpu's in the future, so keep an eye.

1. ChatRWKV v2: with "stream" and "split" strategies. 3G VRAM is enough to run RWKV 14B :)

[ Parent | on HN ]
Re:Open source ChatGPT with LLaMA implementation trains 15x ... by IanCal on Monday February 27, 2023 @06:15PM (#34963394)
You have to split it up which slows it down a lot. The 14B model doesn't fit fully on a 3090, though the 7B fits easily and is very fast. Other replies either may have meant this or thought the original comment was about llama.
[ Parent | on HN ]