News for hackers. Stuff that matters.
Re:Looking for an app where I can chat with a book by vunderba on Saturday March 16, 2024 @07:41PM (#39730450)
There is an AI app for this already:

https://chatfai.com/characters/book

I haven't tried it, but if I had to guess how it is built they're probably just setting up RAG vector databases at a per-book level and then augmenting a given character's context window with information from the vector database relevant to the conversation.

It would be relatively trivial (weekend project) to roll your own using streamlit + quant + ggerganov llama.cpp using Vicuna/Mistral/etc. Hardest part would be separating an entire book into a well representated set of embeddings.

Re:Looking for an app where I can chat with a book by blizzardman on Sunday March 17, 2024 @08:38PM (#39739403)
Thank you for the recommendation. RAG would make sense Here is my understanding of how to do it

1. Use sentence transformer, transform the entire harry potter or lord of the ring book into embeddings

2. transform query into embeddings -> "why don't gandolf sent the eagles"

3. Find most relevant text using ANN through the query embeddings

4. pipe in the context + query to llama

However the result is not very good, am I missing something in RAG?

Re:Looking for an app where I can chat with a book by vunderba on Tuesday March 19, 2024 @12:28AM (#39752946)
Make sure you're using a SOTA embedding model (UAE, embedding-ada-002, etc) that is capable of creating a vector from a reasonably large token size, see here for comparisons: https://huggingface.co/spaces/mteb/leaderboard

Experiment with a "sliding scale" around the book (paragraphs, pages, etc). Try to use a graph to relate book sections, etc.

Consider setting up a tuner with well defined questions and answers to search for optimality around embeddings.