News for hackers. Stuff that matters.
Re:Our next-generation model: Gemini 1.5 by megaman821 on Thursday February 15, 2024 @10:33AM (#39383891)
So, will this outperform any RAG approach as long as the data fits inside the context window?
Re:Our next-generation model: Gemini 1.5 by CuriouslyC on Thursday February 15, 2024 @11:02AM (#39384322)
A perfect RAG system would probably outperform everything in a larger context due to prompt dilution, but in the real world putting everything in context will win a lot of the time. The large context system will also almost certainly be more usable due to elimination of retrieval latency. The large context system might lose on price/performance though.
Re:Our next-generation model: Gemini 1.5 by TheGeminon on Thursday February 15, 2024 @10:40AM (#39384006)
Outperform is dependent on the RAG approach (and this would be a RAG approach anyways, you can already do this with smaller context sizes). A simplistic one, probably, but dumping in data that you don't need dilutes the useful information, so I would imagine there would be at least _some_ degradation.

But there is also the downside of "tuning" the RAG to return less tokens you will miss extra context that could be useful to the model.

Re:Our next-generation model: Gemini 1.5 by megaman821 on Thursday February 15, 2024 @10:48AM (#39384140)
Doesn't their needle/haystack benchmark seem to suggest there is almost no dilution? They pushed that demo out to 10M tokens.
Re:Our next-generation model: Gemini 1.5 by ArcaneMoose on Thursday February 15, 2024 @10:37AM (#39383950)
Cost would still be a big concern
Re:Our next-generation model: Gemini 1.5 by saliagato on Thursday February 15, 2024 @10:38AM (#39383966)
basically, yes. Pinecone? Dead. Azure AI Search? Dead. Quadrant? Dead.
Re:Our next-generation model: Gemini 1.5 by _boffin_ on Thursday February 15, 2024 @10:41AM (#39384027)
Prompt token cost still a variable.
Re:Our next-generation model: Gemini 1.5 by chasd00 on Thursday February 15, 2024 @04:22PM (#39389091)
are you going to upload 10M tokens to Gemini on every request? That's a lot of data moving around when the user is expecting a near realtime response. Seems like it would still be better to only set the context with information relevant to the user's prompt which is what plain rag does.