News for hackers. Stuff that matters.
Re:I want flexible queries, not RAG by bambax on Tuesday May 21, 2024 @06:10PM (#40434674)
Not really. RAG works very differently from a general (or generalist?) LLM.

RAG is vector search first. It encodes the query, finds nearest vectors in the vector database, retrieves the fragments attached to those vectors, and then sends those vectors to the LLM for it to summarize them.

A general LLM like Gemini or Claude or ChatGPT first produces an answer to a question, based on its training. This doesn't involve searching any external source at that point. Then after that answer is produced, the LLM can try to find sources that match what it has come up with.

Re:I want flexible queries, not RAG by simonw on Tuesday May 21, 2024 @08:48PM (#40436119)
You don't have to use vector search to implement RAG. You can use other search mechanisms instead or as well - whatever it takes to populate the context with the material most likely to help answer the user's question.

Two common alternatives to vector search:

1. Ask the LLM to identify key terms in the user's question and use those terms with a regular full-text search engine

2. If the content you are answering questions about is short enough - an employee handbook for example - just jam the whole thing in the context. Claude 3 supports 200,000 tokens and Gemini Pro 1.5 supports a million so this can actually be pretty effective.

Re:I want flexible queries, not RAG by bambax on Wednesday May 22, 2024 @03:15AM (#40438182)
You're right and I was imprecise. 2. is how RAG is implemented in most cases. (I wouldn't call 1. 'RAG' exactly, though?)
Re:I want flexible queries, not RAG by bigfudge on Wednesday May 22, 2024 @04:35AM (#40438714)
HYDE is a related technique. Ask the model to generate a response with no context, then use this for semantic search agains actual data and respond by summarising these documents.
Re:I want flexible queries, not RAG by simonw on Wednesday May 22, 2024 @09:20AM (#40440615)
I'd definitely call 1 (the FTS version) RAG. It's how Bing and Google Gemini and ChatGPT Browse work - they don't have a full vector index of the Web to work with (at least as far as I know), they use the model's best guess at an appropriate FTS query instead.
Re:I want flexible queries, not RAG by outofpaper on Tuesday May 21, 2024 @06:20PM (#40434755)
This is a generalization. These proprietary systems do different things at different times. With GPT4o you can see little icons when a RAG is in use or when code and tests are being used.

People, we have to stop talking about what we know as though it's all there is. Don't confuse our knowledge for understanding. Understanding only comes from repeadly trying to prove our understandings wrong and learning how things truly are.

Re:I want flexible queries, not RAG by acka on Tuesday May 21, 2024 @10:10PM (#40436659)
This. There is no point in arguing amongst ourselves when we're all wearing blindfolds and it is the zookeeper who decides what parts of the proprietary elephant we're allowed to touch.
Re:I want flexible queries, not RAG by ozr on Tuesday May 21, 2024 @10:23PM (#40436728)
That's not what RAG is. RAG is the process of adding relevant information to a prompt in an LLM. It's a form of in-context learning.
Re:I want flexible queries, not RAG by mr_toad on Tuesday May 21, 2024 @11:20PM (#40437001)
It’s right there in the name - first you Retrieve relevant information (often a vector lookup) then you use it to Augment the prompt, then you Generate an answer.

It’s bloody useful if you can’t cram your entire proprietary code base into a prompt.

Re:I want flexible queries, not RAG by kergonath on Wednesday May 22, 2024 @01:16AM (#40437566)
What makes it useful is precisely that it is not learning.
Re:I want flexible queries, not RAG by ozr on Wednesday May 22, 2024 @02:59PM (#40444773)
Yes it is; it's just not updating the weights.

https://ai.stanford.edu/blog/understanding-incontext/