News for hackers. Stuff that matters.

A comment on French AI startup Mistral releases ‘unmoderated’ chatbot via torrent. Parent comment

Re:French AI startup Mistral releases ‘unmoderated’ chatbot ... by thelastparadise on Saturday September 30, 2023 @09:09AM (#37715075)
Could it run on a 4x 3090 24GB rig?

These can be built for about $4500 or less all-in.

Inference FLOPs will be roughly equivalent to ~1.8X A100 perf.

[ Parent | on HN ]
Re:French AI startup Mistral releases ‘unmoderated’ chatbot ... by mutex_man on Saturday September 30, 2023 @09:17AM (#37715149)
You could run it on a single high end GPU. I can run llama2's models ,(except 70b) on my 4080.
[ Parent | on HN ]
Re:French AI startup Mistral releases ‘unmoderated’ chatbot ... by 7moritz7 on Saturday September 30, 2023 @09:43AM (#37715365)
This can run on 1x 2060S 8 GB
[ Parent | on HN ]
Re:French AI startup Mistral releases ‘unmoderated’ chatbot ... by ComputerGuru on Saturday September 30, 2023 @06:02PM (#37720348)
With what degree of quantization?
[ Parent | on HN ]
Re:French AI startup Mistral releases ‘unmoderated’ chatbot ... by 7moritz7 on Saturday October 14, 2023 @08:46AM (#37880055)
None, just the default weights using ollama. It's fast too. 13b is where things get slow
[ Parent | on HN ]
Re:French AI startup Mistral releases ‘unmoderated’ chatbot ... by chriscappuccio on Saturday September 30, 2023 @07:00PM (#37720779)
does a 4x 3090 rig need nvswitch
[ Parent | on HN ]