Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on MistralLite by Amazon Web Services. Parent comment
There are quantizations already. I would suggest running it on KoboldCPP, or any of the UIs recommended on this page: https://huggingface.co/TheBloke/MistralLite-7B-GGUF
Personally, I run the elx2 quantization with exllamav2 because its faster and more coherent in the same amount of vram. You can do that with Aphrodite or Text-gen-ui: https://huggingface.co/models?search=MistralLite%20exl2
There are no comments on this comment yet.
There are quantizations already. I would suggest running it on KoboldCPP, or any of the UIs recommended on this page: https://huggingface.co/TheBloke/MistralLite-7B-GGUF
Personally, I run the elx2 quantization with exllamav2 because its faster and more coherent in the same amount of vram. You can do that with Aphrodite or Text-gen-ui: https://huggingface.co/models?search=MistralLite%20exl2