The False Promise of Open-sourced LLMs context length

Re:The False Promise of Open-sourced LLMs context length by shubhamgrg04 on Friday June 30, 2023 @12:31AM (#36530516)

Isn't condensing rotary embeddings just a clever hack to circumvent the original training limitations? Is it really a sustainable solution for extending context length in language models?

[ Parent | on HN ]

Re:The False Promise of Open-sourced LLMs context length by kaiokendev on Friday June 30, 2023 @01:53AM (#36531030)

I am not sure how to parse this. Can you clarify?

[ Parent | on HN ]

Re:The False Promise of Open-sourced LLMs context length by bioemerl on Friday June 30, 2023 @07:23AM (#36533018)

Yes and no.

Yes, it actually works. That's what matters at the end of the day.

But no, eventually you're going to have to fine tune on something that has a larger context or training a brand new model with the position embeddings semi randomized so that it learns to generalize in the first place instead of needing hacks like this to function.

But training a model costs millions and millions of dollars, so we're going to have to wait until some very generous group decides to do all that training and then release that model open source.

Or releases the model file for a paid charge fee, I'd pay like $200 for a good one.

[ Parent | on HN ]