News for hackers. Stuff that matters.
Re:MistralLite by Amazon Web Services by brucethemoose2 on Wednesday November 01, 2023 @12:35PM (#38100802)
Yeah it works fine. Mistral is a llama architecture llm.

There are quantizations already. I would suggest running it on KoboldCPP, or any of the UIs recommended on this page: https://huggingface.co/TheBloke/MistralLite-7B-GGUF

Personally, I run the elx2 quantization with exllamav2 because its faster and more coherent in the same amount of vram. You can do that with Aphrodite or Text-gen-ui: https://huggingface.co/models?search=MistralLite%20exl2

There are no comments on this comment yet.