News for hackers. Stuff that matters.
Re:The Secret Sauce behind 100K context window in LLMs: all ... by stoniejohnson on Saturday June 17, 2023 @10:43PM (#36376846)
I don't know much, but this isn't surprising based on the little I know.

Transformers predict the next token.

If your question is at the end of the prompt, the start of an answer is a more likely next token than if the question is at the beginning of the prompt followed by a ton of other relevant, but non-question-forming tokens.

Still, if you had to put the question at the beginning of your prompt, a transformer is more likely to give an answer than an RNN.

Re:The Secret Sauce behind 100K context window in LLMs: all ... by jumpCastle on Sunday June 18, 2023 @12:20AM (#36377284)
It is fine tuned to maximize reward though, not likelihood. And it provides an answer in both cases, just not as well.
Re:The Secret Sauce behind 100K context window in LLMs: all ... by stoniejohnson on Sunday June 18, 2023 @12:25AM (#36377311)
So since a model is fine tuned via RLHF my point doesn't stand?

Genuine question; it would be interesting if some other mechanism was at play here.

Re:The Secret Sauce behind 100K context window in LLMs: all ... by jumpCastle on Sunday June 18, 2023 @12:42AM (#36377382)
For an answer I would expect it to get the same reward for both question orderings. So naively I would expect it to not be affected by the ordering.