News for hackers. Stuff that matters.
Re:LLaMA Now Goes Faster on CPUs by claritise on Monday April 01, 2024 @08:38AM (#39893498)
Just create a RAG with wikipedia as the corpus and a low parameter model to run it and you can basically have an instantly queryable corpus of human knowledge runnable on an old raspberry pi.
Re:LLaMA Now Goes Faster on CPUs by CaptainOfCoit on Monday April 01, 2024 @10:06AM (#39894325)
> a low parameter model

> on an old raspberry pi

I bet the LLM responses will be great... You're better off just opening up a raw text dump of Wikipedia markup files in vim.

Re:LLaMA Now Goes Faster on CPUs by boywitharupee on Monday April 01, 2024 @10:20AM (#39894423)
but which model to tokenize with? is there a leaderboard for models that are good for RAG?
Re:LLaMA Now Goes Faster on CPUs by sroussey on Monday April 01, 2024 @01:36PM (#39896727)
“For RAG” is ambiguous.

First there is a leaderboard for embeddings. [1]

Even then, it depends how you use them. Some embeddings pack the highest signal in the beginning so you can truncate the vector, while most can not. You might want that truncated version for a fast dirty index. Same with using multiple models of differing vector sizes for the same content.

Do you preprocess your text? There will be a model there. Likely the same model you would use to process the query.

There is a model for asking questions from context. Sometimes that is a different model. [2]