Re:Full text search on 400M US court cases
by rpedela on Thursday November 19, 2020 @02:45PM
(#25153554)
Elasticsearch, Lucene under the hood, implements an inverted index which is an extremely fast data structure for text search. ES has clustering as a primary feature too and many search features that can significantly improve relevance that you won't find in MySQL and most other databases.
Re:Full text search on 400M US court cases
by devy on Thursday November 19, 2020 @04:02PM
(#25154313)
Have you tried Toshi[1] or MeiliSearch[2]. I wonder how it would compare in terms of operational costs (monthly cloud hosting bill) at the current data size.
Re:Full text search on 400M US court cases
by jabo on Thursday November 19, 2020 @04:52PM
(#25154816)
Do you have the structured dataset somewhere? I’d love to index it in Typesense [1] and see how it does.
I recently tried a 32M songs dataset [2] and it works great, so I’m on the lookout for larger datasets to benchmark with.
Re:Full text search on 400M US court cases
by lolive on Thursday November 19, 2020 @03:59PM
(#25154296)
Plus it does not accept joins. So you basically have to denormalize all your data before injecting into Elastic.
It helps speedup things. But is a headache to manage on a day to day basis.
Re:Full text search on 400M US court cases
by rpedela on Thursday November 19, 2020 @06:07PM
(#25155446)
Yeah. What I do is create a view that does all the joins then the middleware just needs to do "SELECT * FROM my_view". If the DB has good JSON support, I will also convert the data into an ES index request with SQL so the middleware becomes even simpler.
Re:Full text search on 400M US court cases
by lolive on Friday November 20, 2020 @01:20AM
(#25157775)
Let’s say you have a mostly read-only DB (otherwise things are different).
Does it work when the view is insanely big? [i am not an expert at DBs, so my vision of a big DB might amuse you, but let’s say I have millions of rows to assemble as a view].
Re:Full text search on 400M US court cases
by arminiusreturns on Thursday November 19, 2020 @03:05PM
(#25153746)
How would you say it stands up to splunk these days?
Re:Full text search on 400M US court cases
by richardbarosky on Thursday November 19, 2020 @01:59PM
(#25153036)
Elasticsearch is a search platform. A "database" but meant for search stuff. It's not part of MySQL.
Re:Full text search on 400M US court cases
by scottydelta on Friday November 20, 2020 @03:18AM
(#25158374)
The gist is Elasticsearch is a full-index database. Whatever data goes in gets indexed as compared to only indexing certain fields in MySQL on which you perform search frequently. Think of Elasticsearch as MongoDB + full-indexing. It's a document storage with blazing fast search and aggregation.
Sorry, I'm just curious.
It says MySQL 8 and Elasticsearch 7.8. I don't have much experience in elasticsearch, I wanted to know how does elasticsearch makes it faster? Is it like an extension that makes it faster? Or Elasticsearch has its own data store that consumes data from the database and magically makes it faster?
Thanks.
[1]https://www.reddit.com/r/programming/comments/jg4rkv/how_a_s...