I tend to do this with GPT-4 even on the context window in default ChatGPT (or more often I bookend it with instructions). I find it pays off at even 1000 tokens.
Transformers predict the next token.
If your question is at the end of the prompt, the start of an answer is a more likely next token than if the question is at the beginning of the prompt followed by a ton of other relevant, but non-question-forming tokens.
Still, if you had to put the question at the beginning of your prompt, a transformer is more likely to give an answer than an RNN.
Genuine question; it would be interesting if some other mechanism was at play here.
I don't know about the super large contexts but you can also just make the text data clearly delimited instead of putting the query at the end, so that "predict the next token" isn't fighting the instruction-following training
DATA DATA
DATA DATA ...
-- boundary --
QUERY
or the arguably equivalent: QUOTED TEXT
-- boundary --
REPLY / COMMENTARY
The inverse shape is also common: INSTRUCTIONS
-- boundary --
DATA / TEXT ON WHICH TO WORK
For example, most exercise lists and test books are written like that.The somewhat less frequent patterns are more random mix of:
WHAT
-- boundary --
ON WHAT
-- boundary --
WHAT ELSE
-- boundary --
ON WHAT ELSE
-- boundary --
(...)
CLOSING REMARKS
Most of my HN comments are structured like that, for example. Including this one.Boundary here can take many forms. Extra newlines, --, ``` blocks ```, > - prefixed text, and lists (both OL and UL) are all common methods used to structure text, and are seen both in training data and in inference. We know LLM picks up on those structure markets at high-frequency level (e.g. using extra newlines or -- lines to separate distinct blocks seems effective). But I imagine it also picks up on the low-frequency patterns, which is why payload followed by, or bracketed with, instructions is something it "knows" how to process, whereas if you use less common structuring patterns, you're more likely to confuse the LLM.
From what you're saying, it sounds like there is some kind of recency bias in these models.
then I ask to generate. it's very powerful, as it removes the preamble and other chitcat from the response, and empower the system message over what's in the user message.
example: https://i.imgur.com/7fF0CZm.png?maxwidth=123456789&fidelity=... here the first agent message is the one conditioning the answer beginning, and I only generate the second agent.
(sorry mobile user imgur may return a low res unreadable image idk what's the alternative in 2023)
Fixed by adding ------- line between the materials and the question in the end.
I've found Claude to be better than GPT4 at creative writing and explanations, while GPT4 seems to be better at logic-puzzlish stuff.
Anecdotally I do find myself using Claude for summarization. It does seem to require less prompt crafting to get good results so when I just need an article or YouTube video summarized it's nice to be able to just drop it in and be like, "summarize this"
1. https://chrome.google.com/webstore/detail/perplexity-ask-ai/...
Maybe because it's basically impossible to get access to it right now…
Anthropic is a relatively new startup that probably has 3) & 4) from their history at OpenAI. But I don't see how they could have 1) & 2).
OpenAI didn’t build a bespoke supercomputer, but trained on Azure (with a preferential contract thanks to their investors): https://openai.com/gpt-4
> GPT-4 was trained on Microsoft Azure AI supercomputers
Cohere trained on GCP: https://techcrunch.com/2021/11/17/google-cloud-teams-up-with...
> Aidan Gomez, co-founder and CEO at Cohere[:] “We scrape the data to train these big models, we train them on massive TPU pods”
Stability AI trains on AWS: https://aws.amazon.com/blogs/machine-learning/stability-ai-b...
> With Amazon SageMaker, Stability AI will build AI models on compute clusters with thousands of GPU or AWS Trainium chips
Given Anthropic’s recent partnership announcement, they likely train on GCP nowadays: https://www.anthropic.com/index/anthropic-partners-with-goog...
> Anthropic will leverage Google Cloud's cutting-edge GPU and TPU clusters to train, scale, and deploy its AI systems
This would be similar with humans if everything was given verbally.
The huggingface transformers library exposes this as past_key_values, here's an example on GPT-2: https://huggingface.co/docs/transformers/model_doc/gpt2#tran...
Moreover, the inability to cache transformers makes the use of large context windows quite costly, as all previous messages must be sent with each call. In this context, the RWKV-LM project on GitHub (https://github.com/BlinkDL/RWKV-LM) might offer a solution. They claim to achieve performance comparable to transformers using an RNN, which could potentially handle a 100-page document and cache it, thereby eliminating the need to process the entire document with each subsequent query. However, I suspect RWKV might fall short in handling complex tasks that require maintaining multiple variables in memory, such as mathematical computations, but it should suffice for many scenarios.
On a related note, I believe Anthropics' Claude is somewhat underappreciated. In some instances, it outperforms GPT4, and I'd rank it somewhere between GPT4 and Bard overall.