News for hackers. Stuff that matters.
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by pyth0 on Thursday May 11, 2023 @01:41PM (#35905533)
This more or less is already a thing and it's called RAG [1][2]. It essentially allows you to have a database of embeddings (in this case your book) from which a model can pull knowledge from while producing answers. As for the standard operation of these generative models, the context window is the only working memory it has and so it must see the entire text each time.

[1] https://arxiv.org/abs/2005.11401

[2] https://huggingface.co/docs/transformers/model_doc/rag

Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by m1sta_ on Thursday May 11, 2023 @03:37PM (#35907330)
Cam you help me understand this? The research appears to be from a few years ago. Can this be used with Claude (for example)? How is it different to the approach many people are taking with vector stores and embeddings?
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by make3 on Thursday May 11, 2023 @03:47PM (#35907479)
it's not different. RAG is a way to train embedding stores end to end
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by make3 on Thursday May 11, 2023 @09:59PM (#35910965)
somehow got down voted on something I'm a professional expert at
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by pyth0 on Thursday May 11, 2023 @03:54PM (#35907562)
Other people seem to be suggesting that the user would do the retrieval of the relevant parts of the book from a vectordb first, and then feed those sections along with the question as the prompt. Conceptually it is very similar (and it too uses vector database), but with RAG it would happen as part of the inferencing pipeline and therefore achieve better performance than the end user emulating it.
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by ukuina on Friday May 12, 2023 @02:05AM (#35912537)
Yep, but your retrieval from the vector DB becomes your relevancy bottleneck.