News for hackers. Stuff that matters.
Re:Postgres full text search is good enough by pilif on Monday September 29, 2014 @02:21AM (#8381869)
If it's indeed the same document, you're screwed. Yes. But in Postgres, I could at least have different column (or join tables with different language codes) for different languages and then query them with either dictionary.

ElasticSearch doesn't support that: All fields in a document are assumed to be of the same language which leads to a huge duplication of documents in cases where most fields of a document are language-independent.

Re:Postgres full text search is good enough by Argorak on Monday September 29, 2014 @03:01AM (#8381926)
> All fields in a document are assumed to be of the same language

That is not true. You can configure analyzers on a field basis, both for search and index.

Re:Postgres full text search is good enough by pilif on Monday September 29, 2014 @03:10AM (#8381940)
Oh. Cool. Thanks. When I was looking into this, it wasn't yet possible. That's very interesting for me.

ElasticSearch has MUCH better support for language analysis to the point where it might be worth it to move from Postgres tsearch to ElasticSearch, but this limitation was a huge hurdle for me not being able to move.

Re:Postgres full text search is good enough by Argorak on Monday September 29, 2014 @03:23AM (#8381959)
I am surprised, that's possible since any version I've been working with (0.12x and up).

What you cannot do is control that via client (e.g. "this index request uses the chinese analyzer for that field only"), because the analyzer setting in the index request overwrites all other configs. Mappings supported that for a long time.

One of the arts of search optimization on Elasticsearch is finding fields that you want to index multiple times with different analyzers and combine in a clever fashion.

Re:Postgres full text search is good enough by donw on Monday September 29, 2014 @02:23AM (#8381871)
That's basically what I ended up doing -- writing an analyzer that made a reasonable guess as to what language each word was, and filing it into the appropriate bucket, with a generic bucket for "everything else".

Ranking is another story, though.

Re:Postgres full text search is good enough by izacus on Monday September 29, 2014 @02:49AM (#8381915)
I don't know about ElasticSearch, but the more popular alternative Solr, can do that easily by using multiple cores (each configured and tweaked for each language) and doing sharding queries, which merge results for you.
Re:Postgres full text search is good enough by jrochkind1 on Monday September 29, 2014 @08:27AM (#8382575)
you don't need to use multiple cores for that, although you could -- but in general, anything you can do in multiple cores you can do identically in one core, the only reason to use multiple cores is for performance when you have so much corpus that one machine can't handle it.

But yeah, Solr gives you a lot of flexibility, perhaps more than ElasticSearch (which is both it's pro and it's con, for 'on the path' use cases I get the impression ElasticSearch requires less thinking in configuration than solr. although i have not any direct experience with ElasticSearch).