News for hackers. Stuff that matters.
Re:Jina AI Launches First Open-Source 8K Text Embedding, Riv... by simonw on Wednesday October 25, 2023 @11:18PM (#38021267)
I've been getting great results for related documents by embedding entire blog posts, e.g. here: https://til.simonwillison.net/gis/pmtiles#related

I'm not sure how I would do that after chunking.

Re:Jina AI Launches First Open-Source 8K Text Embedding, Riv... by thomasahle on Thursday October 26, 2023 @01:12AM (#38021838)
Did you compare with simple baselines like bag-of-words and word vectors?
Re:Jina AI Launches First Open-Source 8K Text Embedding, Riv... by simonw on Thursday October 26, 2023 @01:16AM (#38021868)
My previous implementation used TF-IDF - I basically took all the words in the post and turned them into a giant "word OR word OR word OR word" search query and piped that through SQLite full-text search. https://til.simonwillison.net/sqlite/related-content

I jumped straight from that to OpenAI embeddings. The results were good enough that I didn't spend time investigating other approaches.

Re:Jina AI Launches First Open-Source 8K Text Embedding, Riv... by rolisz on Thursday October 26, 2023 @01:35AM (#38021995)
That's not quite tfidf though. I agree you can get better results than that with Ada embeddings, but I would argue you can get even better results with embeddings from smaller chunks.
Re:Jina AI Launches First Open-Source 8K Text Embedding, Riv... by simonw on Thursday October 26, 2023 @01:50AM (#38022093)
I guess technically it's bm25, since it's using the rank mechanism in SQLite FTS5: https://www.sqlite.org/fts5.html#sorting_by_auxiliary_functi...
Re:Jina AI Launches First Open-Source 8K Text Embedding, Riv... by thomasahle on Thursday October 26, 2023 @03:42AM (#38022864)
> Into a giant "word OR word OR word OR word"

Does that mean you'd return other docs if they share just one word?

The idea of tfidf is that it gives you a vector (maybe combined with pca or a random dimensionality reduction) that you can use just like an Ada embedding. But you still need vector search.

Re:Jina AI Launches First Open-Source 8K Text Embedding, Riv... by simonw on Thursday October 26, 2023 @09:12AM (#38025063)
My goal for related articles was to first filter to every document that shared at least one word with the target - which is probably EVERY document in the set - but then rank them based on which ones share the MOST words, scoring words that are rare in the corpus more highly. BM25 does that for free.

Then I take the top ten by score and call those the "related articles".