Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu...
by gardnr on Friday December 01, 2023 @06:38AM
(#38485662)
I can fit 13B Q4 K M models on a 12GB RTX 3060. It OOMs when the context window goes above 3k. I get 25 tok/s.
[ Parent
| on HN ]
There are no comments on this comment yet.