News for hackers. Stuff that matters.

A comment on Amazon Elasticsearch Service Is Now Amazon OpenSearch Service. Parent comment

Re:Amazon Elasticsearch Service Is Now Amazon OpenSearch Ser... by _msw_ on Thursday September 09, 2021 @02:25PM (#28472938)
It is pretty clear to me that Elastic is planning to build their ANN features differently than OpenDistro's k-NN implementation, or other plugins modules that extend Easticsearch in similar ways. They now will build on the Apache Lucene capabilities that were collaboratively built "upstream" by a number of individuals, some that work for Amazon and some that work for Elastic.

From the linked issue, it seemed that they were originally planning to develop this as a proprietary feature of Elasticsearch, without contributing the functionality to Apache Lucene, but then changed direction when the Apache Lucene developers (some of which are currently employed to do such work by Amazon) started to build its approximate nearest neighbor (ANN) vector search capabilities. [1]

It's great to see folks that work for Elastic collaborating and building on what is in Apache Lucene to extend the utility of ANN with Hierarchical Navigable Small World Graphs (HNSW) [2]! From this, I think it should be possible to implement an Open Source version of the functionality with a compatible API, if that is something that OpenSearch users seek.

[1] https://issues.apache.org/jira/browse/LUCENE-9004

[2] https://github.com/apache/lucene/pull/250

[ Parent | on HN ]
Re:Amazon Elasticsearch Service Is Now Amazon OpenSearch Ser... by binarymax on Friday September 10, 2021 @10:50AM (#28482362)
"it should be possible to implement an Open Source version of the functionality".

Elasticsearch is still open source...just not in a license that Amazon can take advantage of.

[ Parent | on HN ]