News for hackers. Stuff that matters.
Re:Microsoft Phi-2 model changes licence to MIT by moffkalast on Saturday January 06, 2024 @05:12AM (#38890147)
Close is a subjective term. Vicuna-33B from over half a year ago gets within 22 ELO of 3.5 in the arena leaderboard, but in practice the refusals are reducing 3.5 and other RLHFd models' ratings a lot and they're not even close.

You can try Phi-2 with wasm here, I mostly just get gibberish out of it though: https://huggingface.co/spaces/radames/Candle-phi1-phi2-wasm-...

Mixtral is the only model that properly matches 3.5 at the moment.

Re:Microsoft Phi-2 model changes licence to MIT by Y_Y on Saturday January 06, 2024 @06:43AM (#38890622)
The Elo scoring system is named after a Hungarian man called Arpad Elo (a simplification of his original more Hungarian name). My phone helpfully miscorrects it to "ELO" probably because it prefers Jeff Lynne's Electric Light Orchestra. Anyway,

Elo is a proper name, not an acronym!

Re:Microsoft Phi-2 model changes licence to MIT by moffkalast on Saturday January 06, 2024 @07:10AM (#38890788)
TIL, interesting. I always figured it must be some kind of abbreviation.
Re:Microsoft Phi-2 model changes licence to MIT by stavros on Saturday January 06, 2024 @05:22AM (#38890197)
That makes sense, thank you. There seems to be some inflation, as Mistral is supposed to be GPT-3.5-level, and Mixtral is supposed to be nearer GPT-4, but yeah, it sounds suspicious in practice, even though Mistral is very good.
Re:Microsoft Phi-2 model changes licence to MIT by moffkalast on Saturday January 06, 2024 @05:38AM (#38890270)
Well in some things it totally can be to some extent, yes. You can almost certainly get a Mistral 7B fine tuned for a specific thing (e.g. coding) and it will likely be about as good as 3.5 in that specific thing (not a super high bar in objective terms). For all the other areas it may suffer in performance relative to its original self, but for some applications that's fine. As for GPT-4 it's about 120 ELO points [0] above Mixtral, and that's even the distilled turbo version. Not even close imo, especially when Mixtral is far less censored.

Both 3.5 and 4 have changed drastically over the past year with continued fine tuning, quantization, etc. so what some people consider their level is not exactly a fixed point either.

[0] The actual leaderboard I'm referencing, it has its biases but it's the most generally indicative thing available right now: https://chat.lmsys.org

Re:Microsoft Phi-2 model changes licence to MIT by regularfry on Saturday January 06, 2024 @06:03AM (#38890387)
Mixtral is only 32 ELO ahead of the best 7B model on that leaderboard, although I suspect that might be understating the difference.