News for hackers. Stuff that matters.

A comment on ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model. Parent comment

Re:ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based... by Dylan16807 on Thursday January 19, 2023 @11:14PM (#34449172)
If it fits in system memory, is it still faster on GPU than CPU? Does that involve swapping out one layer at a time? Otherwise I'm very curious how it handles the PCIe latency.

Enough system memory to fit 84GB isn't all that expensive...

[ Parent | on HN ]
Re:ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based... by tempay on Friday January 20, 2023 @01:02AM (#34449736)
Yes, the connection between system memory and the GPU isn’t fast enough to keep the compute units fed with data to process. Generally PCIe latency isn’t as much of a problem as bandwidth.
[ Parent | on HN ]