Re:Our next-generation model: Gemini 1.5
by megaman821 on Thursday February 15, 2024 @10:33AM
(#39383891)
So, will this outperform any RAG approach as long as the data fits inside the context window?
[ Parent
| on HN ]
Re:Our next-generation model: Gemini 1.5
by CuriouslyC on Thursday February 15, 2024 @11:02AM
(#39384322)
A perfect RAG system would probably outperform everything in a larger context due to prompt dilution, but in the real world putting everything in context will win a lot of the time. The large context system will also almost certainly be more usable due to elimination of retrieval latency. The large context system might lose on price/performance though.
Re:Our next-generation model: Gemini 1.5
by TheGeminon on Thursday February 15, 2024 @10:40AM
(#39384006)
Outperform is dependent on the RAG approach (and this would be a RAG approach anyways, you can already do this with smaller context sizes). A simplistic one, probably, but dumping in data that you don't need dilutes the useful information, so I would imagine there would be at least _some_ degradation.
But there is also the downside of "tuning" the RAG to return less tokens you will miss extra context that could be useful to the model.
Re:Our next-generation model: Gemini 1.5
by megaman821 on Thursday February 15, 2024 @10:48AM
(#39384140)
Doesn't their needle/haystack benchmark seem to suggest there is almost no dilution? They pushed that demo out to 10M tokens.
Re:Our next-generation model: Gemini 1.5
by ArcaneMoose on Thursday February 15, 2024 @10:37AM
(#39383950)
Cost would still be a big concern
Re:Our next-generation model: Gemini 1.5
by saliagato on Thursday February 15, 2024 @10:38AM
(#39383966)
basically, yes. Pinecone? Dead. Azure AI Search? Dead. Quadrant? Dead.
Re:Our next-generation model: Gemini 1.5
by _boffin_ on Thursday February 15, 2024 @10:41AM
(#39384027)
Prompt token cost still a variable.
Re:Our next-generation model: Gemini 1.5
by chasd00 on Thursday February 15, 2024 @04:22PM
(#39389091)
are you going to upload 10M tokens to Gemini on every request? That's a lot of data moving around when the user is expecting a near realtime response. Seems like it would still be better to only set the context with information relevant to the user's prompt which is what plain rag does.