News for hackers. Stuff that matters.
Re:Investing in new vector database development vs enhancing... by brigadier132 on Saturday November 25, 2023 @06:21PM (#38417703)
why do companies use databases like elastic search when they could just use inverted indexes in postgres?

Performance, features, and scalability

Re:Investing in new vector database development vs enhancing... by simonw on Saturday November 25, 2023 @06:38PM (#38417844)
I use Elasticsearch along with PostgreSQL, but I don't think of Elasticsearch as a database. It's an external index that I denormalize some of my data into.
Re:Investing in new vector database development vs enhancing... by tyre on Sunday November 26, 2023 @12:11AM (#38419335)
ES is a good parallel to a vector database.

run a search against ES => hydrate records from postgres

run a search against a vector database => hydrate LLM context from documents

Neither is a primary canonical datastore. Postgres is for ES and S3 is for documents.

Re:Investing in new vector database development vs enhancing... by riku_iki on Saturday November 25, 2023 @11:15PM (#38419118)
so, its just happened that there is no good implementation of full text search in postgres, but fundamentally nothing prevents from implementing such search, except horizontal scalability if it is needed.
Re:Investing in new vector database development vs enhancing... by johnthescott on Saturday November 25, 2023 @11:41PM (#38419218)
the pg extension called "rum" is very performant on millions of pdfs.

https://github.com/postgrespro/rum

Re:Investing in new vector database development vs enhancing... by riku_iki on Sunday November 26, 2023 @12:11AM (#38419334)
my case is tens of billions of short docs, and default search is fine except index creation is single threaded, so it takes multiple days to build index, and github page says rum is even slower..
Re:Investing in new vector database development vs enhancing... by aforwardslash on Sunday November 26, 2023 @04:19AM (#38420285)
I love Postgres, but you're comparing apples and oranges. You don't have the same level of query sophistication neither the same indexing quality you get from eg. elasticsearch. As a quick example, the other day I got a bug in PostgreSQL because "searching" does not perform diacritics conversion. Also, stuff like stop words, similarity search and whatnot either required dedicated indexes or are downright "round hole square peg" problems.
Re:Investing in new vector database development vs enhancing... by mkarrmann on Sunday November 26, 2023 @10:12AM (#38422162)
I believe they're being rhetorical, and agree that there are good reasons to use elasticsearch over postgres.