Zenfetch (https://www.zenfetch.com/) does an awesome job at this!

the ability to do this is pretty accessible nowadays, its just a little expensive for normal use for the layman

There is a nice opensource extension that does exactly that: https://github.com/memex-life/memex

I tried to use it, and it just does not work very well, I don't it's because implementation is bad, it's just RAG (retrieval based generation) does not work well outside of some simple use cases.

What are those simple use cases, and where do you see Retrieval Augmented Generation fall over?

I think it works better if query is a larger chunk of text. Like, if you have an email from a customer and want to compose a response based on some relevant documentation, it should work well.

But for a use case where you want to retrieve something from browsing history you would mainly use a short search query, just few words. in this case embeddings are too ambiguous and relevance of retrieved content is not great.

That’s not a problem with RAG itself that’s an issue with your retriever. In the original RAG paper they used two vanilla BERT models and cosine similarity but there’s no requirement you do that. Use any retriever that gets you high precision. Use BM25 if you want, it’s simple and cheap.

You’re right in saying there’s not enough semantic meaning in the text of the query. The domain of queries and the domain of documents are very different. That’s why a real retrieval system will train the query encoder and doc encoder to be closer in their embedding space using click data. This is what Google is doing.

"train the query encoder and doc encoder to be closer in their embedding space using click data" <- Any papers/resources you know where I can learn more about this process?

Triplet loss.

https://pytorch.org/docs/stable/generated/torch.nn.TripletMa...

Triplet loss takes an anchor, positive, and negative. In this case the anchor is your query, the positive is a similar doc, and the negative is a dissimilar doc. When you train, backpropagate the loss to both the doc and the query encoder.

alway wonder why browsers don't keep at least a text version of the pages for search

nowadays I expect browsers to incorporate small LLMs like Mistral out of the box

if history search worked better then you would visit google/bing less, so I expect pushback from browser vendors

also: https://bugs.chromium.org/p/chromium/issues/detail?id=297648

From a comment on that issue page, dating a little over a decade ago:

> Shouldn't this be left to users to decide whether they want a feature or not.

Yes, those were good times. Nowadays, it's anathema to software vendors, and increasingly even to open source devs.

Another advertising revenue win for Query Jerry from marketing.

Not every browser vendor runs a search engine…

Apple and Mozilla aren’t Google. Apple runs their own searches on the App Store and iPhone, but Google pays them to be default. There’s no reason Apple couldn’t also index your search history in a more meaningful way other than perhaps space requirements.

Opera used to do this! And then when you searched in the top bar, it would also search your visited pages. It was very useful.

Hope the Raindrop.io [1] founder (u/johngreen) sees this! Tangentially, quite a bit of what I read ends up as comments here, and it can be a pickle remembering where I read something to properly cite ("what keywords do I need to use to find the piece I'm thinking of"). Anything that reduces the human effort in citing a thesis (accurately!) is welcome.

[1] https://raindrop.io/

That gets real messy real fast!