A promising and potentially useful approach for adding additional context to your prompts.

How likely is it that context size will greatly increase in the coming year or two? Are there fundamental limits, or could we reasonably expect greatly increased context size in the future?

We saw an almost 10x jump between GPT-3.5-turbo and gpt-4 (4k to 32k tokens).

But are we likely to get another 10x jump with say GPT4.5, and 10x over that with GPT5?

Or is context likely to stay fixed around 32k for the foreseeable future?

That is the question.

If you look into this into more detail:

A) people have had a hard time reproducing it, and

B) more damning, the "compressed" version uses more tokens than the original (https://gist.github.com/VictorTaelin/d293328f75291b23e203e9d...)

> In practice, plain, well-designed summaries should be optimal to fit larger documents in the context.

> This concept has potential, though; building lookup tables seems to outperform long text summarization.

It's a clever idea, and I agree that lookup tables and external storage of memory is likely going to be important at some point, but I suspect that's going to come out of giving LLM more ability to externally reference "long-term" memory rather than compressing everything into immediate context.

People are smashing the duplos together too hard.

Sessions are getting treated as more valuable than necessary.

This should look more like functions.

Needs lower prices and greater availability for that.

But until then… smashing duplos together.

I spent some time on this, here is a better version: https://mobile.twitter.com/gfodor/status/1643415357615640577

This is the most disappointing part about the frenzy.

Obvious ideas being held up as something more than that.

If you think this is incredible, give yourself some time to go play with LLMs. Try things. Clever things. Stupid things.

It helps set a more reasonable scale for these sensational sounding snippets.

This post from @gfodor is also a really interesting read on the topic of compressing prompts, not sure what the main use case would be though!

https://twitter.com/gfodor/status/1643444605332099072?s=20

ChatGPT 3.5 in response to a brief analysis of the community gave out: "It seems like the conversation you provided is discussing the idea of compressing prompts or summaries of larger documents in order to fit them into a smaller context. While the idea has potential, it appears that people have had difficulty reproducing it and that lookup tables and external storage of memory may be more effective in the long term. Additionally, there is discussion about the possibility of greatly increasing context size in the future, with the example given that there was an almost 10x jump between GPT-3.5-turbo and GPT-4 in terms of token capacity."

ZipGPT is a novel, LLM-based non-deterministic compression algorithm. Give our beta version a try. The results may surprise you!

The compressed one is shorter number of characters but does it have fewer tokens? https://platform.openai.com/tokenizer

One reply points out that even though the text length is 1/4 of the original, the “compressed” output is actually LONGER in token count!

https://twitter.com/dogeofcoin/status/1642918892602290179

A more accurate title would be "decrease GPT4's context size by asking it to obfuscate your prompts". You're getting a prompt that consumes more tokens and isn't particularly faithful to the original.

The context length includes all of GPT4's replies, is that correct? If I want to minimize the context length, should I be asking GPT4 to limit its replies to just the fix that is requested?

GPT4 seems to prefer long-winded replies, i.e., when I ask for what amounts to a one-line fix, it repeats the entire block of code with the one line correction. In contrast, Replit's Ghostwriter often gives a concise reply showing only the one-line fix, and I have to ask for the entire block when it isn't clear where the fix is applied.