News for hackers. Stuff that matters.
Re:MistralLite by Amazon Web Services by hfjjbf on Wednesday November 01, 2023 @12:48PM (#38101028)
This account has prepackaged and quantized models for everything, really awesome.

https://huggingface.co/TheBloke

Re:MistralLite by Amazon Web Services by brucethemoose2 on Wednesday November 01, 2023 @01:00PM (#38101218)
Indeed.

But TheBloke doesn't (yet) do exl2 quantization. That one is almost better done yourself, as the bpw target is arbitrary, and requires profiling text that should probably fit your intended use (though many just use wikitext as a good default).