News for hackers. Stuff that matters.
Re:Microsoft Phi-2 model changes licence to MIT by ranguna on Saturday January 06, 2024 @04:08AM (#38889825)
This model has been in the top for quite a while, what's so good about it?
Re:Microsoft Phi-2 model changes licence to MIT by intellectronica on Saturday January 06, 2024 @04:16AM (#38889861)
Excellent performance for this model size and inference cost. Best model you can run on a device a small as a phone and get performance close to GPT-3.5 level.

The structure and the training data are also interesting - sparse model using curated synthetic data to achieve much better accuracy than is achieved in models trained on random internet text.

Re:Microsoft Phi-2 model changes licence to MIT by stavros on Saturday January 06, 2024 @04:58AM (#38890068)
Is it really close to GPT-3.5 at 2.7B?
Re:Microsoft Phi-2 model changes licence to MIT by regularfry on Saturday January 06, 2024 @05:11AM (#38890136)
It's 2.7B, not 1.1. In my experience it goes off the rails and starts generating nonsense after a few paragraphs, but I haven't dug too much into tweaking the kv cache params to see if that's controllable. It also needs a fair bit of prompt massaging to get it to do what you want. So no, not GPT3.5, but it's comfortably better than anything else in its size class.
Re:Microsoft Phi-2 model changes licence to MIT by realPtolemy on Saturday January 06, 2024 @06:36AM (#38890588)
How is it compared to 7B LLaMA quantized to run on a raspberry pi?
Re:Microsoft Phi-2 model changes licence to MIT by jasonjmcghee on Saturday January 06, 2024 @11:19PM (#38898393)
If you can run quantized 7B, nothing beats mistral and its fine tunes- like openhermes2.5
Re:Microsoft Phi-2 model changes licence to MIT by stavros on Saturday January 06, 2024 @05:20AM (#38890183)
This sounds much more realistic, thanks!
Re:Microsoft Phi-2 model changes licence to MIT by eurekin on Saturday January 06, 2024 @02:03PM (#38894219)
What are kv cache params?
Re:Microsoft Phi-2 model changes licence to MIT by regularfry on Saturday January 06, 2024 @07:13PM (#38896980)
Key-value cache in the attention layers. There was a paper a little while back about how maintaining the first N tokens across an extended context helped an LLM keep sane for longer, and it turns out you can replicate it with the right CLI arguments to llama.cpp.
Re:Microsoft Phi-2 model changes licence to MIT by WiSaGaN on Saturday January 06, 2024 @05:36AM (#38890254)
In my experience, it's great at its size, but obviously worse than mistral:7b-instruct-v0.2. Currently mixtral:8x7b-instruct-v0.1 is the lowest inference cost model at similar performance level of GPT-3.5.
Re:Microsoft Phi-2 model changes licence to MIT by moffkalast on Saturday January 06, 2024 @05:12AM (#38890147)
Close is a subjective term. Vicuna-33B from over half a year ago gets within 22 ELO of 3.5 in the arena leaderboard, but in practice the refusals are reducing 3.5 and other RLHFd models' ratings a lot and they're not even close.

You can try Phi-2 with wasm here, I mostly just get gibberish out of it though: https://huggingface.co/spaces/radames/Candle-phi1-phi2-wasm-...

Mixtral is the only model that properly matches 3.5 at the moment.

Re:Microsoft Phi-2 model changes licence to MIT by Y_Y on Saturday January 06, 2024 @06:43AM (#38890622)
The Elo scoring system is named after a Hungarian man called Arpad Elo (a simplification of his original more Hungarian name). My phone helpfully miscorrects it to "ELO" probably because it prefers Jeff Lynne's Electric Light Orchestra. Anyway,

Elo is a proper name, not an acronym!

Re:Microsoft Phi-2 model changes licence to MIT by moffkalast on Saturday January 06, 2024 @07:10AM (#38890788)
TIL, interesting. I always figured it must be some kind of abbreviation.
Re:Microsoft Phi-2 model changes licence to MIT by stavros on Saturday January 06, 2024 @05:22AM (#38890197)
That makes sense, thank you. There seems to be some inflation, as Mistral is supposed to be GPT-3.5-level, and Mixtral is supposed to be nearer GPT-4, but yeah, it sounds suspicious in practice, even though Mistral is very good.
Re:Microsoft Phi-2 model changes licence to MIT by moffkalast on Saturday January 06, 2024 @05:38AM (#38890270)
Well in some things it totally can be to some extent, yes. You can almost certainly get a Mistral 7B fine tuned for a specific thing (e.g. coding) and it will likely be about as good as 3.5 in that specific thing (not a super high bar in objective terms). For all the other areas it may suffer in performance relative to its original self, but for some applications that's fine. As for GPT-4 it's about 120 ELO points [0] above Mixtral, and that's even the distilled turbo version. Not even close imo, especially when Mixtral is far less censored.

Both 3.5 and 4 have changed drastically over the past year with continued fine tuning, quantization, etc. so what some people consider their level is not exactly a fixed point either.

[0] The actual leaderboard I'm referencing, it has its biases but it's the most generally indicative thing available right now: https://chat.lmsys.org

Re:Microsoft Phi-2 model changes licence to MIT by regularfry on Saturday January 06, 2024 @06:03AM (#38890387)
Mixtral is only 32 ELO ahead of the best 7B model on that leaderboard, although I suspect that might be understating the difference.
Re:Microsoft Phi-2 model changes licence to MIT by jasonjmcghee on Saturday January 06, 2024 @01:13PM (#38893679)
Close to gpt-3.5? Because I’ve tried it and fine tuned variants and it’s been horrible. Next to useless on general tasks. No where near mistral 7 and absolutely not even close to gpt-3.5.

Best 2.7b? sure.

I have always thought of this model as something you fine tune for a very specific task or dataset.

For others that don’t strongly disagree with the parent comment, can you point me to some examples?