Re:70B Llama 2 at 35tokens/second on 4090
by rwmj on Wednesday September 13, 2023 @06:00AM
(#37494627)
If you need to use all the memory then swapping breaks down, best you can do is try to divide the work so half can be done, then you completely swap all the memory and do the other half. Also PCIe is (relatively) slow for moving gigabytes of data back and forth.
[ Parent
| on HN ]
There are no comments on this comment yet.