News for hackers. Stuff that matters.

A comment on PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU [pdf].

Re:PowerInfer: Fast Large Language Model Serving with a Cons... by Havoc on Tuesday December 19, 2023 @07:54PM (#38704176)
Really clever trick. GPU/CPU splits are currently painfully slow so this may just make them more bearable.
[ Parent | on HN ]

There are no comments on this comment yet.