thirdtrigger

Might be interesting to add an optional embedded Weaviate [1] with a flat-index [2] to the project. It wouldn't use external services and is fully disk-based. Would allow you to search the whole filesystem (about 1.5kb per file (384 dimensions) which would be added to the metadata as well).

1. https://weaviate.io/developers/weaviate/installation/embedde... 2. https://weaviate.io/developers/academy/py/vector_index/flat

TL;DR (from the article)

How the HammerBot Works

We also want to share details about how HammerBot works on the backend. Right now, our developers are using OpenAI’s text-davinci-003 model, otherwise known as GPT-3, trained on a custom dataset of our articles. The data is stored in a vector database from Weaviate and the bot is coded primarily with Python, using LangChain, a framework that makes it easy to customize AI output.

When you enter a prompt, the server queries against the dataset that’s stored in Weaviate to get the search results. Those are then sent to the LLM to help it develop a consistent response to your question.

A lot of chatbots will talk to you about anything on Earth. But don’t ask HammerBot for a knitting pattern! It’s designed to have limits on what it will talk about; it focuses on the expertise you can only get from Tom’s Hardware, so it may say it doesn’t know or can’t answer if prompts that fall outside of its training.

Agreed – one can also use Weaviate which comes with an OOTB OpenAI module leveraging the embeddings end-point https://weaviate.io/developers/weaviate/current/retriever-ve...

There is also an integration with Weaviate.

* Docs: https://weaviate.io/developers/weaviate/current/retriever-ve... * Post: https://weaviate.io/blog/2022/12/Cohere-multilingual-with-we...

Thanks for your kind words about Weaviate (I’m affiliated).

What you propose is certainly possible, you could do that with a vector database like Weaviate (https://weaviate.io). Import the data, vectorize it, and connect it to a person with cross-references

+ Weaviate for vector based search. Has a BSD-3 license. https://weaviate.io/developers/weaviate/current/

> At this point vector search is almost 'solved'

Well, there is is still a lot to do. Going from memory to disk, etc. But I see your point.

What might be a bit confusing is the mix of vector search libraries with vector search engines. It's a bit like comparing an inverted index library with SOLR :)

But we (I.e., the wider vector search ecosystem) are working on this

Agreed with your observations about the difference between the libraries and databases. Maybe for a follow-up article? It's nice to see some independent research on the topic, tho.

Agreed with fzliu, you can also use https://weaviate.io (disclaimer, I'm affiliated with Weaviate). You might also like this article which describes why one might want to use a vector search engine: https://db-engines.com/en/blog_post/87

Im affiliated with a company that makes one. We create a vector search engine called Weaviate [1] we also publish content on how it's done and the search engine itself is open source which might be helpful for you too.

[1] https://weaviate.io

Always when Ted Nelson shows up, this point is made one way or the other. I even once heard Vint Cerf make these points at a conference.

1. Xanadu being vaporware 2. His ego

What I really don't get is what Ted Nelson does to trigger this anger in people.

E.g., his book computer lib/dream machines inspired so many, and I find him super self-aware. His appearance in Werner Herzog’s movie about the internet was (IMHO) so refreshing.

I love Ted's work, and it's a huge inspiration to me in my daily work. This video just adds to that.

I'm affiliated with Weaviate, so maybe nice to get this out here for the record :)

We call Weaviate a "vector search engine" (i.e., we prefer "vector search engine" because it describes the type of database) since around Aug, 2020

Github: https://github.com/semi-technologies/weaviate/tree/a3967aff5...

The reason was simple; our community started to say that the mixed vector and scalar filter search capabilities were what they liked most.

Also, our benchmarks are available for quite some time here: https://weaviate.io/developers/weaviate/current/benchmarks/a...

They are based on ann-benchmarks.com but adjusted for full databases.

Interesting writeup. I'm of the opinion that the problem of the naming issue (how to call "things"?) sits in the idea that going from structured documents to structured data is one abstraction level too deep (i.e., people don't agree on how to call "things"). I believe this can be solved by similarity search; if we can approximate the data and represent the structure in embeddings. Hopefully, this might be a step in the 2nd try, as mentioned in the MD :)

> It would be like wikipedia, but even more all encompassing, and far more transformational.

You might like to see this (https://weaviate.io/developers/weaviate/current/tutorials/se...) as a step in this direction because it contains the structured Wikipedia data and the embeddings to target individual nodes in the graph.