Re:Mistral 7B,The complete Guide of the Best 7B model
by tayo42 on Tuesday October 31, 2023 @01:02PM
(#38087961)
wow didnt know that, thats really cool, ill have to play with it
[ Parent
| on HN ]
Re:Mistral 7B,The complete Guide of the Best 7B model
by brucethemoose2 on Tuesday October 31, 2023 @02:09PM
(#38088970)
If you want to host for the AI Horde, Aphrodite has the most GPU throughput, and Koboldcpp is the easiest and quite flexible:
https://github.com/PygmalionAI/aphrodite-engine https://github.com/LostRuins/koboldcpp
Picking a model to host is quite a rabbit hole, but I am very interested in SciPhi and Zephyr at the moment.
I havent messed with Petals yet either, but swarm finetuning is very appealing... If thats even implemented atm.