News for hackers. Stuff that matters.
Re:Microsoft Phi-2 model changes licence to MIT by stavros on Saturday January 06, 2024 @05:22AM (#38890197)
That makes sense, thank you. There seems to be some inflation, as Mistral is supposed to be GPT-3.5-level, and Mixtral is supposed to be nearer GPT-4, but yeah, it sounds suspicious in practice, even though Mistral is very good.
Re:Microsoft Phi-2 model changes licence to MIT by moffkalast on Saturday January 06, 2024 @05:38AM (#38890270)
Well in some things it totally can be to some extent, yes. You can almost certainly get a Mistral 7B fine tuned for a specific thing (e.g. coding) and it will likely be about as good as 3.5 in that specific thing (not a super high bar in objective terms). For all the other areas it may suffer in performance relative to its original self, but for some applications that's fine. As for GPT-4 it's about 120 ELO points [0] above Mixtral, and that's even the distilled turbo version. Not even close imo, especially when Mixtral is far less censored.

Both 3.5 and 4 have changed drastically over the past year with continued fine tuning, quantization, etc. so what some people consider their level is not exactly a fixed point either.

[0] The actual leaderboard I'm referencing, it has its biases but it's the most generally indicative thing available right now: https://chat.lmsys.org

Re:Microsoft Phi-2 model changes licence to MIT by regularfry on Saturday January 06, 2024 @06:03AM (#38890387)
Mixtral is only 32 ELO ahead of the best 7B model on that leaderboard, although I suspect that might be understating the difference.