Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and...
by brucethemoose2 on Monday September 04, 2023 @05:32PM
(#37385298)
^
I didn't mean to imply splitting llama up between machines (though that is a thing with llama.cpp), but a pool of clients and servers who make requests and process them:
A few users with half decent PCs can serve a much larger group of people, and the "lesser" hosts can host smaller models to "earn" access to larger ones.
I didn't mean to imply splitting llama up between machines (though that is a thing with llama.cpp), but a pool of clients and servers who make requests and process them:
https://lite.koboldai.net/
A few users with half decent PCs can serve a much larger group of people, and the "lesser" hosts can host smaller models to "earn" access to larger ones.