News for hackers. Stuff that matters.
Re:RAG Using Unstructured Data and Role of Knowledge Graphs by sroussey on Thursday January 18, 2024 @01:57AM (#39038817)
Nice, I’ll look that up.

I was thinking in terms of RAG and turning text into keywords. Any thoughts there?

Re:RAG Using Unstructured Data and Role of Knowledge Graphs by laminarflow027 on Thursday January 18, 2024 @02:53PM (#39046729)
By this I presume you mean build a search index that can retrieve results based on keywords? I know certain databases use Lucene to build a keyword-based index on top of unstructured blobs of data. Another alternative is to use Tantivy (https://github.com/quickwit-oss/tantivy), a Rust version of Lucene, if building search indices via Java isn't your cup of tea :)

Both libraries offer multilingual support for keywords, I believe, so that's a benefit to vector search where multilingual embedding models are rather expensive.