News for hackers. Stuff that matters.
Re:Uncomfortable AWS Truths by idnefju on Sunday September 15, 2019 @08:45PM (#20980314)
Is ES one of these necessary evils? I have been looking for a product similar to it, but it seems ES is always recommended.
Re:Uncomfortable AWS Truths by cbetti on Sunday September 15, 2019 @09:32PM (#20980471)
No, ES is an affordable entry level search engine in a very expensive product space.

But there is nothing new in the standalone search engine space that I am aware of. Many engines have been gobbled up and integrated into larger product suites (see: Oracle Secure Enterprise Search).

Re:Uncomfortable AWS Truths by falcolas on Sunday September 15, 2019 @10:51PM (#20980744)
The problem with ElasticSearch is that it does multiple things at once. It indexes documents. It stores documents. It searches indexes. It serves documents. The indexing and searching should be separated from the storage and serving at any kind of reasonable scale.

But, ES makes it “just work” at the proof of concept and low volume stages, and you can’t easily back out of it when you reach its limitations.

Re:Uncomfortable AWS Truths by castillar76 on Monday September 16, 2019 @02:44PM (#20987243)
For all that people (rightfully) give them grief for being expensive, this is something Splunk got right. You can put your entire Splunk infrastructure on one host: like stem cells, a single Splunk box can perform any of the required functions (ingest, index, search). But you can also separate out each layer into separate hosts and then cluster them, so you have clusters of ingesters feeding clusters of indexers, accessed by clusters of search heads. That should be the architecture ES aims for, where people are given the flexibility to take whichever function is currently a pain point and break it out to a separate set of nodes.
Re:Uncomfortable AWS Truths by pas on Monday September 16, 2019 @03:59AM (#20981961)
Well, ES spent many years polishing itself, but there is one new very rough kid on the block:

https://github.com/toshi-search/Toshi/blob/master/README.md

Re:Uncomfortable AWS Truths by sidlls on Sunday September 15, 2019 @10:22PM (#20980638)
ES is one of those technologies that "everyone" goes to when they want any kind of basic text search indexing "at scale" (usually with a delusional notion of what their scaling needs actually are), because text search/indexing is tedious and hard at any scale that a basic DB query would fall over on (which is generally a much bigger scale than many want to admit).
Re:Uncomfortable AWS Truths by mooreds on Sunday September 15, 2019 @08:54PM (#20980343)
I saw a post about this and the other open source alternative was solr.

Not sure about support/community though.

Re:Uncomfortable AWS Truths by ink_13 on Sunday September 15, 2019 @09:19PM (#20980426)
It's funny because under the hood, they're both Lucene.
Re:Uncomfortable AWS Truths by mooreds on Monday September 16, 2019 @04:35PM (#20988296)
I guess that's the power of open source, right?
Re:Uncomfortable AWS Truths by 33degrees on Sunday September 15, 2019 @09:44PM (#20980513)
I don’t find ES that bad, considering its power, but I haven’t had the best experience with the AWS service. Could be user error though....
Re:Uncomfortable AWS Truths by BoorishBears on Sunday September 15, 2019 @10:05PM (#20980580)
I don't fault ES for this necessarily, but the biggest problem with ES is, it's easy to build something that looks like it works, right until it doesn't.

This is a problem for most DBs. You put stuff in, you take stuff out, and it works... but you're putting it in the DB in a way that will make the queries you need nearly impossible to do within your real-world/production constraints (response time, resources required, etc.)

The thing is ES is so different from most DBs, so you need to relearn how to do things right. And to make matters a little more difficult, all DBs have a lot of knobs, but ES just has way more knobs you need to know how to turn to get something that won't fall on it's face in production.

I don't remember my experience with ES on AWS that well besides it was behind on major versions for a while at one point, and if you set up ES resource with CloudFormation... god help you if you need to roll them back.

Using serverless with ES looked something like

api-1 api-2-because-1-is-frozen-rolling-back-cloudformation-es-instances api-3-because-2-is-also-frozen-rolling-back-cloudformation-es-instances api-4-...

Re:Uncomfortable AWS Truths by shiredude95 on Sunday September 15, 2019 @10:19PM (#20980624)
Have you tried elasticcloud? They solve a lot of access problems present in AWS Elasticsearch along with providing the most recent version for deployment.
Re:Uncomfortable AWS Truths by Redsquare on Monday September 16, 2019 @02:07AM (#20981492)
Elasticcloud has terrible security features - they removed IP whitelisting.
Re:Uncomfortable AWS Truths by t0mbstone on Monday September 16, 2019 @04:50PM (#20988473)
They do have user/password authentication though, which is something that barebones ElasticSearch doesn't normally have out of the box by default.