News for hackers. Stuff that matters.
Re:Full text search on 400M US court cases by rpedela on Thursday November 19, 2020 @02:45PM (#25153554)
Elasticsearch, Lucene under the hood, implements an inverted index which is an extremely fast data structure for text search. ES has clustering as a primary feature too and many search features that can significantly improve relevance that you won't find in MySQL and most other databases.
Re:Full text search on 400M US court cases by devy on Thursday November 19, 2020 @04:02PM (#25154313)
Have you tried Toshi[1] or MeiliSearch[2]. I wonder how it would compare in terms of operational costs (monthly cloud hosting bill) at the current data size.

[1]: https://news.ycombinator.com/item?id=18895655

[2]: https://news.ycombinator.com/item?id=22685831

Re:Full text search on 400M US court cases by jabo on Thursday November 19, 2020 @04:52PM (#25154816)
Do you have the structured dataset somewhere? I’d love to index it in Typesense [1] and see how it does.

I recently tried a 32M songs dataset [2] and it works great, so I’m on the lookout for larger datasets to benchmark with.

[1] https://news.ycombinator.com/item?id=22181437

[2] https://songs-search.typesense.org/

Re:Full text search on 400M US court cases by lolive on Thursday November 19, 2020 @03:59PM (#25154296)
Plus it does not accept joins. So you basically have to denormalize all your data before injecting into Elastic. It helps speedup things. But is a headache to manage on a day to day basis.
Re:Full text search on 400M US court cases by rpedela on Thursday November 19, 2020 @06:07PM (#25155446)
Yeah. What I do is create a view that does all the joins then the middleware just needs to do "SELECT * FROM my_view". If the DB has good JSON support, I will also convert the data into an ES index request with SQL so the middleware becomes even simpler.
Re:Full text search on 400M US court cases by lolive on Friday November 20, 2020 @01:20AM (#25157775)
Let’s say you have a mostly read-only DB (otherwise things are different).

Does it work when the view is insanely big? [i am not an expert at DBs, so my vision of a big DB might amuse you, but let’s say I have millions of rows to assemble as a view].

Re:Full text search on 400M US court cases by arminiusreturns on Thursday November 19, 2020 @03:05PM (#25153746)
How would you say it stands up to splunk these days?