News for hackers. Stuff that matters.
Re:Microsoft Phi-2 model changes licence to MIT by regularfry on Saturday January 06, 2024 @05:11AM (#38890136)
It's 2.7B, not 1.1. In my experience it goes off the rails and starts generating nonsense after a few paragraphs, but I haven't dug too much into tweaking the kv cache params to see if that's controllable. It also needs a fair bit of prompt massaging to get it to do what you want. So no, not GPT3.5, but it's comfortably better than anything else in its size class.
Re:Microsoft Phi-2 model changes licence to MIT by realPtolemy on Saturday January 06, 2024 @06:36AM (#38890588)
How is it compared to 7B LLaMA quantized to run on a raspberry pi?
Re:Microsoft Phi-2 model changes licence to MIT by jasonjmcghee on Saturday January 06, 2024 @11:19PM (#38898393)
If you can run quantized 7B, nothing beats mistral and its fine tunes- like openhermes2.5
Re:Microsoft Phi-2 model changes licence to MIT by stavros on Saturday January 06, 2024 @05:20AM (#38890183)
This sounds much more realistic, thanks!
Re:Microsoft Phi-2 model changes licence to MIT by eurekin on Saturday January 06, 2024 @02:03PM (#38894219)
What are kv cache params?
Re:Microsoft Phi-2 model changes licence to MIT by regularfry on Saturday January 06, 2024 @07:13PM (#38896980)
Key-value cache in the attention layers. There was a paper a little while back about how maintaining the first N tokens across an extended context helped an LLM keep sane for longer, and it turns out you can replicate it with the right CLI arguments to llama.cpp.