News for hackers. Stuff that matters.
Re:Show HN: I made an app to use local AI as daily driver by discordance on Wednesday February 28, 2024 @07:27AM (#39537005)
RAG:

1. First you create embeddings from your documents

2. Store that in a vector db

3. Ask what the user wants and do a search in the vector db (cosine similarity etc)

4. Feed the relevant search results to your LLM and do the usual LLM stuff with the returned embeddings and chunks of the documents

Re:Show HN: I made an app to use local AI as daily driver by bigfudge on Wednesday February 28, 2024 @08:56AM (#39537941)
Although RAG is often implemented via vector databases to find 'relevant' content, I'm not sure that's a necessary component. I've been doing what I call RAG by finding 'relevant' content for the current prompt context via a number of different algorithms that don't use vectors.

Would you define RAG only as 'prompt optimisation that involves embeddings'?

Re:Show HN: I made an app to use local AI as daily driver by eevmanu on Wednesday February 28, 2024 @03:00PM (#39542766)
Sure thing, your RAG approach sounds intriguing, especially since you're sidestepping vector databases. But doesn't the input context length cap affect it? (chatgpt plus at 32K [0] or gpt4 via open ai at 128K [1]) Seems like those cases would be pretty rare though.

[0]: https://openai.com/chatgpt/pricing#:~:text=8K-,32K,-32K

[1]: https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turb...

Re:Show HN: I made an app to use local AI as daily driver by bigfudge on Friday March 01, 2024 @03:22AM (#39559750)
Yes, context window is a limiting factor, but that's true however you identify the content to augment generation.