News for hackers. Stuff that matters.
Re:Is Microsoft trying to commit suicide? by simonw on Wednesday June 05, 2024 @11:24AM (#40586029)
Answering my own question (with the help of replies on Twitter: https://twitter.com/simonw/status/1798368111038779610 )

Best guess is that there's some level of semantic analysis going on beyond just OCR, which tips the balance (at least for me) to being an "AI" feature based on loose current usage of that term.

The clue is that in this demo https://www.youtube.com/watch?v=aZbHd4suAnQ&t=1062s they show searching for "blue pantsuit with sequin lace" finds something that was described as "peacock" in the text. It looks like this is based on embedding search against image embeddings.

And... one of the 3 SQLite databases created by Recall is called "SemanticImageStore" - which suggests to me there may be a CLIP-style image embeddings model being run on-device here.

Those databases contain "diskann" columns which are likely a reference to Microsoft's vector indexing library: https://github.com/microsoft/DiskANN

Re:Is Microsoft trying to commit suicide? by osolo on Wednesday June 05, 2024 @11:59AM (#40586558)
My team worked on this feature and I can confirm that yes, semantic search is being done for both text and images. This allows you to do fuzzy searches like you mentioned in your comment and use words to match images. Everything runs on your device (all the models, the vector database, etc.) in order to preserve privacy.
Re:Is Microsoft trying to commit suicide? by simonw on Wednesday June 05, 2024 @12:30PM (#40586966)
That's really impressive. Can you share which embedding models you are using for this? Also, is Phi-3 or Phi-3 Vision involved?
Re:Is Microsoft trying to commit suicide? by osolo on Wednesday June 05, 2024 @04:11PM (#40589943)
I don't think they've announced publicly what models we're using. I don't think there's any particular reason for this, but just in case, I can't name them here. I'll see if this can be addressed in a blog post of something.

I can tell you that Recall isn't using Phi. Rather, it's using a collection of models that are much more tuned (and therefore much more efficient) for the feature.