News for hackers. Stuff that matters.
Re:The Secret Sauce behind 100K context window in LLMs: all ... by littlestymaar on Saturday June 17, 2023 @10:16PM (#36376711)
Isn't that weird? I mean weren't transformers/attention explicitly designed to avoid this problem faces by RNNs?
Re:The Secret Sauce behind 100K context window in LLMs: all ... by stoniejohnson on Saturday June 17, 2023 @10:43PM (#36376846)
I don't know much, but this isn't surprising based on the little I know.

Transformers predict the next token.

If your question is at the end of the prompt, the start of an answer is a more likely next token than if the question is at the beginning of the prompt followed by a ton of other relevant, but non-question-forming tokens.

Still, if you had to put the question at the beginning of your prompt, a transformer is more likely to give an answer than an RNN.

Re:The Secret Sauce behind 100K context window in LLMs: all ... by jumpCastle on Sunday June 18, 2023 @12:20AM (#36377284)
It is fine tuned to maximize reward though, not likelihood. And it provides an answer in both cases, just not as well.
Re:The Secret Sauce behind 100K context window in LLMs: all ... by stoniejohnson on Sunday June 18, 2023 @12:25AM (#36377311)
So since a model is fine tuned via RLHF my point doesn't stand?

Genuine question; it would be interesting if some other mechanism was at play here.

Re:The Secret Sauce behind 100K context window in LLMs: all ... by jumpCastle on Sunday June 18, 2023 @12:42AM (#36377382)
For an answer I would expect it to get the same reward for both question orderings. So naively I would expect it to not be affected by the ordering.
Re:The Secret Sauce behind 100K context window in LLMs: all ... by furyofantares on Sunday June 18, 2023 @12:30AM (#36377341)
If you've got 20 tokens of query at the start and then 200 tokens of text data that it's querying, it seems really impressive that it's able to work out (via instruct tuning) to answer the query rather than continue the text data. A continuation of the text data is the actual most likely next token.

I don't know about the super large contexts but you can also just make the text data clearly delimited instead of putting the query at the end, so that "predict the next token" isn't fighting the instruction-following training

Re:The Secret Sauce behind 100K context window in LLMs: all ... by TeMPOraL on Sunday June 18, 2023 @04:34AM (#36378372)
IDK. Ignoring the "transformers predict the next token" statement, which feels at best technically correct but missing the point, I imagine this comes down to the network learning "low-frequency" patterns in the training data. That is, in both training and instruct fine-tuning, the model is likely to encounter text structured like:

  DATA DATA
  DATA DATA ...
  -- boundary --
  QUERY
or the arguably equivalent:

  QUOTED TEXT
  -- boundary --
  REPLY / COMMENTARY
The inverse shape is also common:

  INSTRUCTIONS
  -- boundary --
  DATA / TEXT ON WHICH TO WORK
For example, most exercise lists and test books are written like that.

The somewhat less frequent patterns are more random mix of:

  WHAT
  -- boundary --
  ON WHAT
  -- boundary --
  WHAT ELSE
  -- boundary --
  ON WHAT ELSE
  -- boundary --
  (...)
  CLOSING REMARKS
Most of my HN comments are structured like that, for example. Including this one.

Boundary here can take many forms. Extra newlines, --, ``` blocks ```, > - prefixed text, and lists (both OL and UL) are all common methods used to structure text, and are seen both in training data and in inference. We know LLM picks up on those structure markets at high-frequency level (e.g. using extra newlines or -- lines to separate distinct blocks seems effective). But I imagine it also picks up on the low-frequency patterns, which is why payload followed by, or bracketed with, instructions is something it "knows" how to process, whereas if you use less common structuring patterns, you're more likely to confuse the LLM.