News for hackers. Stuff that matters.
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by behnamoh on Friday January 12, 2024 @11:06AM (#38969713)
Despite what people say about ϕ-2, I never liked its responses. It clearly lacks depth and consistency.
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by eightysixfour on Friday January 12, 2024 @11:33AM (#38970133)
I feel like Phi-2 is all about the context of the fact that it is a <3b parameter model. It is incredible in that context, it isn't incredible in others.
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by make3 on Friday January 12, 2024 @12:02PM (#38970654)
it's definitely what it's about. it's insanely strong for its size
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by yufeng66 on Friday January 12, 2024 @01:17PM (#38971749)
Phi-2 basically demonstrated that you don't need a very large model to figure out language. It not very smart but speaks perfect English. it's not obvious the best way to gain IQ is to have a larger language model. some other structure might be needed.
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by behnamoh on Friday January 12, 2024 @01:52PM (#38972248)
But isn't that something that even smaller GPT-2 models demonstrated already?
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by erichocean on Saturday January 13, 2024 @01:25AM (#38977781)
No, and this is covered in the various Phi papers, as well as TinyStories [0]:

> Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words even after extensive training.

[0] https://arxiv.org/abs/2305.07759

Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by visarga on Friday January 12, 2024 @11:53AM (#38970497)
maybe it's useful for fine-tuning jobs, not for free prompting

I have had a similar bad experience with it trying to prompt it

Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by m3kw9 on Friday January 12, 2024 @12:31PM (#38971091)
It’s a model for low rM usage 4gigs? I heard it would work well for RAG to speak to your docs, instead of relying purely on its parameters to answer.
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by coder543 on Friday January 12, 2024 @01:42PM (#38972101)
Did you try Dolphin Phi-2? The Dolphin fine-tune seems better to me in this case.
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by valine on Friday January 12, 2024 @12:41PM (#38971250)
The method they use is surprisingly simple. They claim GPTs can’t effectively generate beyond the context window because our models overfit on positional encodings. The fix is literally to cap the positional encodings at inference time.

It makes sense intuitively that the exact position of tokens really only matters for adjacent or near adjacent tokens. For far away tokens a rough position is fine.

Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by nulld3v on Friday January 12, 2024 @01:43PM (#38972109)
RoPE scaling has been a thing for a while already: https://arxiv.org/abs/2306.15595

Does anybody know what the difference is between the approach in OP vs other RoPE scaling approaches?

Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by hexaga on Friday January 12, 2024 @02:20PM (#38972622)
The core problem is: there's not enough unique, trained positions. Naively going past the end of training ctx makes you run straight into out of distribution positions, and things become incoherent.

For a model trained with a ctx size of 2, that looks like: `[0, 1, *incoherence starts* 2, 3]`

Existing RoPE scaling methods try to stay in-distribution by assigning positions between the known in-distribution ones: `[0, 0.5, 1, 1.5]`. This is still ~kinda OOD, but works w/ some fine tuning.

The method in OP breaks the core premise that we need unique positions at all, and just gives multiple tokens the same position: `[0, 0, 1, 1]`.

Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by cma on Friday January 12, 2024 @03:05PM (#38973243)
Wouldn't that mean (until higher level embeddings) compound phrases far away are unordered? And numbers are fragmented by token boundary and scrambled up?
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by valine on Friday January 12, 2024 @03:47PM (#38973789)
Embeddings at lower layers aren’t going to be looking very far beyond nearby or adjacent embeddings as they refine their meaning. For a number like 3.14, the tokens 3 and 14 are important to each other, but entirely unimportant to understand the meaning of a question later in the context. It’s only at later layers that an embedding representing the concept of PI becomes important to the question embeddings.

As I understand it the positional encodings are calculated relative to the token in question. It’s not like 3 and 14 are unordered tokens from their own perspective.

Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by scosman on Friday January 12, 2024 @01:35PM (#38972002)
Phi-2 and TinyLlama are so so so impressive for being < 3B parameter models. They can run on a phone, and are pretty snappy.

Benchmarks: https://github.com/ggerganov/llama.cpp/discussions/4508

I don't see them taking over general purpose chat/query use cases, but fined tuned to a specific use case and embedded into mobile apps, might be how we see LLMs jump from cool tech demos to something that's present in most products.

Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by te_chris on Friday January 12, 2024 @03:07PM (#38973263)
Has anyone successfully fine tuned it for function calling? Thinking could use a lightweight model like this to interpret args in a pipeline then format them for passing down the chain
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by aluxian on Tuesday January 16, 2024 @03:08PM (#39018377)
Thought the same. Ollama has a JSON mode, it's the closest I got to using it for function calling:

  brew install ollama
  brew services start ollama
  curl http://localhost:11434/api/generate -d '{
    "model": "phi:2.7b-chat-v2-q5_K_M",
    "prompt": "Generate a JSON object that models a list of products",
    "raw": true, "format": "json",
    "stream": false
  }' | jq
If you want to keep in touch / share intel: linkedin.com /in/ aluxian or `me` at my username dot com