News for hackers. Stuff that matters.

A comment on Mistral "Mixtral" 8x7B 32k model [magnet]. Parent comment

Re:Mistral "Mixtral" 8x7B 32k model [magnet] by sp332 on Friday December 08, 2023 @11:42AM (#38570993)
I don't see any code in there. What runtime could load these weights?
[ Parent | on HN ]
Re:Mistral "Mixtral" 8x7B 32k model [magnet] by brucethemoose2 on Friday December 08, 2023 @11:54AM (#38571139)
Its presumably llama just like Mistral.

Everything open source is llama now. Facebook all but standardized the architecture.

I dunno about the moe. Is there existing transformers code for that part? It kinda looks like there is based on the config.

[ Parent | on HN ]
Re:Mistral "Mixtral" 8x7B 32k model [magnet] by jasonjmcghee on Friday December 08, 2023 @12:40PM (#38571748)
Mistral is not llama architecture.

https://github.com/mistralai/mistral-src

[ Parent | on HN ]
Re:Mistral "Mixtral" 8x7B 32k model [magnet] by brucethemoose2 on Friday December 08, 2023 @12:47PM (#38571836)
Its basically llama architecture, all but drop in compatible with llama runtimes.
[ Parent | on HN ]
Re:Mistral "Mixtral" 8x7B 32k model [magnet] by refulgentis on Friday December 08, 2023 @12:40PM (#38571749)
Because it's JSON? :)
[ Parent | on HN ]