News for hackers. Stuff that matters.

A comment on Open source ChatGPT with LLaMA implementation trains 15x faster. Parent comment

Re:Open source ChatGPT with LLaMA implementation trains 15x ... by swyx on Monday February 27, 2023 @11:49AM (#34958686)
lol good luck running a 13B model on a single GPU
[ Parent | on HN ]
Re:Open source ChatGPT with LLaMA implementation trains 15x ... by ddren on Monday February 27, 2023 @12:38PM (#34959403)
Seeing the performance of implementations like FlexGen [1], I don't think it would be entirely unreasonable to run a 13B model on a single GPU for personal usage purposes. You are not going to a run a public service out of it, but it probably would be good enough to run your own ChatGPT or Copilot locally.

[1]: https://github.com/FMInference/FlexGen

[ Parent | on HN ]
Re:Open source ChatGPT with LLaMA implementation trains 15x ... by visarga on Monday February 27, 2023 @11:58AM (#34958819)
You need a RTX 3090 24gb
[ Parent | on HN ]