News for hackers. Stuff that matters.
Re:Find anything fast with Google's vector search technology by thirdtrigger on Tuesday December 14, 2021 @02:50PM (#29556554)
Interesting – we are working on an open source vector search engine called Weaviate and did the same for the complete Wikipedia and Wikidata.

[1] Docs: https://www.semi.technology/developers/weaviate/current/

[2] Github: https://github.com/semi-technologies/weaviate

[3] Wikipedia demo dataset: https://github.com/semi-technologies/semantic-search-through...

[4] Wikidata dataset: https://github.com/semi-technologies/biggraph-wikidata-searc...

Last week there was also a feature on Techcrunch about vector search and Weaviate: https://techcrunch.com/2021/12/11/2246180/

Re:Find anything fast with Google's vector search technology by CShorten on Tuesday December 14, 2021 @03:15PM (#29556917)
I've made some videos on Weaviate as well (Henry AI Labs) if interested:

[1] Wikipedia Vector Search Demo with Weaviate: https://www.youtube.com/watch?v=IGB8vjCuay0

[2] Vector Search through Wikidata with Weaviate: https://www.youtube.com/watch?v=T4zlvknSbGc

[3] Demonstrations of Deep Learning: https://www.youtube.com/watch?v=5jbneytoKi0

[4] Weaviate's GraphQL API for Neurosymbolic Search: https://www.youtube.com/watch?v=K_2X48Tln9U

[5] Introducing the Weaviate Vector Search Engine: https://www.youtube.com/watch?v=AS_2U_INpKk

Re:Find anything fast with Google's vector search technology by thirdtrigger on Tuesday December 14, 2021 @03:21PM (#29557005)
These are all great!

There is also this video about modern search engines and Weaviate on the AI Coffee Break YT channel: https://www.youtube.com/watch?v=YkK5IKgxp-c

Re:Find anything fast with Google's vector search technology by visarga on Wednesday December 15, 2021 @03:01AM (#29563196)
Well done promo video.
Re:Find anything fast with Google's vector search technology by mariushn on Wednesday December 15, 2021 @10:02AM (#29566843)
Looks great!

Dumb question: How does Weaviate know that "Scandinavian" is close to "Finnish" ? The source not having "Scandinavian" at all. If their vectors are close, then the "vectorization" is quite standard for any text, and also per language?

Re:Find anything fast with Google's vector search technology by etiennedi on Wednesday December 15, 2021 @10:45AM (#29567536)
The models that create the vector embeddings are trained on either general or domain specific knowledge. So, to oversimplify it a bit: The model has learned - based on the training data it was presented with - that "Scandinavian" has a relationship to "Finnish". Since the vector space is high-dimensional you can think of each language concept having a distinct place in that space. In this case the concept for "Scandinavian" and "Finnish" were close enough that you got a matching result. To simplifiy it even more: The vectors do not represent the words but the meaning behind them. So, the two sentences "I like wine" and "The fermented juice of grapes is my favorite beverage" have zero keywords overlapping, but are semantically identical. So a good model would give them very similar vectors even though a traditional search engine would find zero resemblence between them.

EDIT: Just realized I didn't answer the second part of your question. Yes, the models are language-specific, but there are also multilingual models that work across a large no. of different languages.

Re:Find anything fast with Google's vector search technology by mariushn on Wednesday December 15, 2021 @10:53AM (#29567673)
Thank you! Now I understand why others mentioned that generating vectors is the hard part.
Re:Find anything fast with Google's vector search technology by etiennedi on Wednesday December 15, 2021 @11:20AM (#29568005)
I agree, but at the same time now is the easiest it's ever been to create great vectors. Sentence-Bert [1] by Nils Reimers is a collection of pre-trained models specficially trained to create good vectors. You can use them out of the box with Weaviate. All you have to do is select your desired model [2] and your text (or images, etc.) will be translated into vectors at import time. As I mentioned in another comment, with Weaviate the goal is to make it as easy to use as any existing search engine or database while still providing you the benefits of Deep Learning & Vector Search.

[1] Sentence-BERT: https://sbert.net

[2] Weaviate Customizer with Out-of-the-box models: https://www.semi.technology/developers/weaviate/current/gett...

Re:Find anything fast with Google's vector search technology by mravl on Tuesday December 14, 2021 @04:23PM (#29558030)
Real eyeopener. this will change the search industry completely
Re:Find anything fast with Google's vector search technology by detaro on Tuesday December 14, 2021 @05:00PM (#29558572)
why?
Re:Find anything fast with Google's vector search technology by thirdtrigger on Tuesday December 14, 2021 @05:02PM (#29558614)
That's a fair question – but I'm going to assume the open-source nature is being meant with this.