This is Amazon's fork of Elasticsearch, to avoid the licensing restrictions introduced when they switched from Apache 2 license to "Elastic License and Server Side Public License" - https://www.elastic.co/pricing/faq/licensing

And my bet is it's the one most are going to be using from now on. I used to think this was a fairly black and white issue, but now two things have coloured it for me.

Firstly, dick moves like this: https://github.com/elastic/elasticsearch-py/pull/1623

Secondly, I don't buy the argument from Elastic any more. Yes, the ethical thing to do when you're making money from someone's work is at least contribute back. At the same time though, they're making money from packaging it up and selling it _as a service_. That "as a service" part is where they're making the bucks.

A bonus thirdly; OpenSearch really is Open Source, and ElasticSearch no longer is.

FD: I have a friend who works at Elastic, though he doesn't really colour my opinions of things.

> Firstly, dick moves like this: https://github.com/elastic/elasticsearch-py/pull/1623

I understand that this is unpopular, but you can make a very strong argument that it's to prevent weird errors in the future. I'm also guilty of littering my code with Asserts to ensure the universe is working fine.

The alternative is to allow it to work and then you end up with weird issues like when you connect mysql client to mariadb server (and vice-versa): https://stackoverflow.com/questions/50169576/mysql-8-0-11-er...

> Secondly, I don't buy the argument from Elastic any more. Yes, the ethical thing to do when you're making money from someone's work is at least contribute back. At the same time though, they're making money from packaging it up and selling it _as a service_. That "as a service" part is where they're making the bucks.

That's just an opinion, yes they have a service, and yes it competes with Amazon. Is it cool for Amazon to take a body of work and sell it without supporting it? Are amazon actually supporting it? Is it the same as Elastic using Lucene? (not really because Elastic submits a the majority of fixes to Lucene, but, you get it).

it's kinda gray, I'm sure Amazon thinks they're the good guy, but it's hard for me to look at Elastic as the bad guy in all this.

Amazon did contribute patches back to elasticsearch. Just not as much as Elastic wanted I guess? On the other hand I do think Amazon was probably in the wrong to call their hosted service Elasticsearch.

It seems especially egregious that AWS claimed their hosted offering was a partnership with Elastic when they launched it: https://twitter.com/kimchy/status/1351534454154543106

Breaking backwards compatability for clients would normally be a massive risk, but Elastic took the risk. It severs the link to OpenSearch, totally. We have 2 ecosystems, now.

We switched to OpenSearch from Elastic, as we have an open source platform. It's actually better than open source elastic - with the security plugin and k-nn plugin (an embedding store):

https://www.hopsworks.ai/post/open-distro-for-elasticsearch-...

We’re looking to migrate from Elastic to OpenSearch for these reasons. We’re a really huge elastic user.

Also the EOL support on elastic versions are pretty aggressive now. It’s been a challenge for us to keep up with it.

Totally agree, and that PR seems pointless as AWS OpenSearch has an advanced configuration option that makes OS pretend to be ES for backward compatibility so not really sure what they're achieving with it.

Maybe don't use an overly generous license if you want to open source, but not really...

I doubt it. Their forum seems pretty dead, Elasticsearch has the known name, etc etc. Seems a bit like MariaDb/Mysql where Mysql are still the default choice for most companies (over MariaDb).

I know some people on hackernews etc are upset with the decisions of Elastic. Suspect most people couldn't care less, even if they are even aware.

> Seems a bit like MariaDb/Mysql where Mysql are still the default choice for most companies (over MariaDb).

[citation needed]. Some Linux distros (notably Debian, Fedora, Arch) only have MariaDB in their repos and will push you towards it, even if you try to ask for MySQL.

https://mariadb.com/kb/en/distributions-which-include-mariad... and look for “replaces” or “defaulted to”.

CentOS even just installs MariaDB when you `yum install MySQL`

I wonder how much money AWS is making off Elasticsearch? The efforts they’re going through seem to be substantial, which makes me wonder just how much it must be.

Alternatively, it must be a strategic play of some kind with the technology behind it, but that seems very unlikely to me.

I'd doubt there's huge money being made by Amazon, but if they give a profit cut to one or two companies that make products AWS offer as a hosted service, then a bunch more will probably also start changing their licensing to match. Putting resources into building an alternative that discourages this is probably not as much as it might appear (there are already plenty of developers at Amazon, and it's unlikely they're all working on other things that need to take priority), so it may be a good investment.

My company uses it and the bill isn't small. I can imagine that other thousands of customers provide substantial money as well.

Amazon's thing is the network effect, not margins on a particular feature.

and tbh it's enough for most of the people. and opensearch supports also OpenID/Oauth.

> OpenSearch is a registered trademark of Amazon Web Services.

Hidden away at the bottom of the page. It's too bad there isn't an Open Source License that protects smaller business from the tech Goliaths.

Elastic-the-company have over 200 million USD in revenue per quarter and its growing. It's not a small company that needs protection.

well, relative to aws they are a small company ;-)

Only semi-related, but I've recently started using https://www.meilisearch.com/. It's relatively limited, but works great for small use cases. It's also pretty easy to operate. I'm hoping as it continues to grow it will support more features and use cases. I don't think the creators intend to address the same depth of complex features in ElasticSearch (and the like), but that's a desirable attribute in my opinion.

I’ve been using https://typesense.org for the same reasons. They’re not taking investor money so I have more hopes they won’t start throwing everything behind the hosted version over timeS

I can't say enough nice things about Typesense. The team is super responsive, the hosted version is performant and affordable. There's some features missing when compared to other similar projects. But nothing terribly show stopping.

I dislike the precedent that AWS/OpenSearch sets/reinforces.

It’s basically tacit approval that “AWS should be allowed to swing in and destroy your business and you can and should do nothing in response”, it lets AWS (or anyone else who wants to try this) continue with this kind of parasitic behaviour.

> It’s basically tacit approval that “AWS should be allowed to swing in and destroy your business and you can and should do nothing in response”, it lets AWS (or anyone else who wants to try this) continue with this kind of parasitic behaviour.

It's up to open source authors to protect themselves and stop using permissive licenses. There will certainly be a shift away from MIT and co in the future.

There aren't really restrictive open source licenses since any restriction you would actually want for business protection makes it not Open Source.

I have zero problem with companies making proprietary software and we should call it as such. If the way your business makes money is by selling or hosting your software then, ya know, not a good fit for OSS -- totally fine. If you develop a piece of software because it solves a problem for you or your business and you think it might help others and you want community contributions then OSS is for you.

> There aren't really restrictive open source licenses since any restriction you would actually want for business protection makes it not Open Source.

Technically true, practically false; AGPL is 100% FOSS and FAANG won't touch it.

Nope, AGPL is already compromised by FAANG too. See how Mongodb switched from AGPL to SSPL

https://lwn.net/Articles/768670/

> Technically true, practically false; AGPL is 100% FOSS and FAANG won't touch it.

Just offer a DUAL FOSS and commercial license. Who cares what FAANG use, FAANG have stopped significantly contributing back to open source.

Just don’t make your software FOSS. Like you reap what you sow when you release your software as an open source project as a growth hack but when you realize you can’t make money giving your product and then selling hosting when it isn’t your core competency.

Like I can’t be mad at AWS for actually keeping ES alive as a community project for the benefit of the people who actually use and deploy it instead of some freemium company product where shit like security is a paid proprietary add-on.

It doesn't have to destroy your business. There are other ways to make money off FOSS other than competing with cloud providers.

Elastic could provide consulting, support, premium tools with proprietary licenses...

Honestly elasticsearch is kind of a mess these days, I’m pretty excited to see where Amazon takes it. Hopefully we all have a better, truly open-source search engine to use in the future.

They should throw it away and build something new.

I am no fan of a lot of the design choices or the effort required to operate large ES clusters, but there are easily several hundred man-years of development effort behind ES. It would be a really long time before a team starting from scratch could produce something better.

Check out some of the things going on in the Rust space MeiliSearch and QuickWit are 2 options emerging (targeting slightly different use cases), and Tantivy -the Rust equivalent of Lucene- is looking really good.

Quickwit CEO here. For people interested, we are focusing on immutable data (logs of all kinds typically). https://github.com/quickwit-oss/quickwit

We make it much 10x cheaper and easier to scale and operate. We store the index on Amazon S3.

I can’t say I disagree. Lucene is a beautiful, scientific thing; but ES is an increasingly-complex mess of opaque API’s and broken promises. Search engines are really, really hard. There is so much room for something better, more full-featured, more tunable, more reasonable.

How is it a mess?

Feature bloat, bad documentation, API models that are nearly inscrutable. Backwards-incompatible changes are constant, requiring reindexing.

I’d like to see the architecture re-thought to include an indexer, like Apache Druid. I’d like to see a proper command line tool (‘searchctl’?), and API clients a-la gRPC (to be less Java-centric). I’d generally like the project to go back to basics and design for 2022 applications.

On a related note, Druid is hardly a good example of API and good deployment and docs story.

I want to be clear here. I tried to replace Elastic search with OpenSearch. However, besides it's just s fork I couldn't run in. OpenSearch requires a lot of configurations that basically are automatic on ElasticSearch. I had spent so many time trying to fix bugs just to run in on my local machine that made just give up and keep with ElasticSearch.

Given that this is Amazon's fork of ES, and given the terrible experience I've had in the past trying to run Amazon stuff locally, I do wonder if this is somewhat intentional. Although a more reasonable explanation is probably just that easy config is low on amazon's priority list since they want you using their managed version.

Opensearch is Elasticsearch with OpenDistro..

Its not “more” or “less” configuration.

If you mean “paid elasticsearch cloud” is easier to use - I would agree because its well.. a paid service.

After maintaining 27 OSS ES clusters that had to be migrated to OS because OSS ES does not exist anymore, you pretty much have no other option than to use OS, even for security patches only..

This may have been improved recently. I downloaded and ran it on my Linux laptop and it worked instantly. the config file was self-documenting.

Notably, Magento 2.4.4 (due for release in a few days) adds OpenSearch support.

The ElasticSearch dependency has always frustrated me, not least because it broke in annoying ways. OpenSearch could be good news.

Note that OpenSearch is a fork of Elasticsearch, so many (most?) things you might have disliked about ES are likely still there.

Oh, of course. But an open source implementation of ElasticSearch is (based on my experience of open source generally) likely to end up in core distributions again, and have longer support lifetimes/more backports. Plus it makes it easier for SaaS platforms in a lot of ways.

Based on my latest analysis, OpenSearch is very active as the following shows:

https://oss.gitsense.com/insights/github?q=pull-age%3A%3C%3D...

but it is still not as active as Elasticsearch, which is to be expected given how long it has been around as the following shows:

https://oss.gitsense.com/insights/github?q=pull-age%3A%3C%3D...

Full disclosure: The above insights are from my tool

activity alone is not a reliable proxy for quality

This is true, but it is a huge proxy for investment and survivability. In the beginning, it was hard to tell how much Amazon would invest in OpenSearch, but it appears now (based on activity) that Amazon is committed to OpenSearch.

Based on my experience, I can see it. I had the OpenSearch PM reach out to us because of our work with OpenSearch. AWS are investing for the long haul, here.

Yeah if you look at only changes to Java files in OpenSearch (OpenSearch + Security) and Elasticsearch, the number of contributors is growing at a steady pace and who knows, it may well surpass Elasticsearch in a year or two.

     month  |    project    | authors | commits | files |  churn  
   ---------+---------------+---------+---------+-------+---------
    2022-03 | elasticsearch |      61 |     529 |   948 |   39307
    2022-03 | opensearch    |      20 |     138 |   603 |   26042
    2022-02 | elasticsearch |     110 |    2783 |  3546 |  287818
    2022-02 | opensearch    |      45 |     582 |  1441 |  126678
    2022-01 | elasticsearch |     126 |    2431 |  3609 |  360828
    2022-01 | opensearch    |      40 |     175 |   640 |   26902
    2021-12 | elasticsearch |     103 |    1803 |  4050 |  301715
    2021-12 | opensearch    |      33 |     166 |   703 |   30182
    2021-11 | elasticsearch |      94 |    1430 |  2309 |  141589
    2021-11 | opensearch    |      31 |     119 |   932 |   44234
    2021-10 | elasticsearch |     102 |    1013 | 12589 | 1217275
    2021-10 | opensearch    |      28 |     142 |  5776 | 1256157
    2021-09 | elasticsearch |      83 |     729 |  2269 |  119506
    2021-09 | opensearch    |      17 |      53 |   126 |    8336
    2021-08 | elasticsearch |      87 |     654 |  2479 |  180385
    2021-08 | opensearch    |      23 |      42 |   128 |    3610

agree that activity is a proxy for (community support) + investment and survivability