Re:How Google understands language like a 10-year-old
by user24 on Monday October 18, 2010 @10:13AM
(#1803556)
who says they're using POS tagging to perform the IR? I've only ever seen it used as a query refinement tool.
I agree, the bag of words / vector space model approach is (from what I've read) hard to beat, especially for a one-size-fits-all tool like google where you can assume nothing about the domain. (Unless they're switching approaches depending on the domain, which would be fascinating)
Re:How Google understands language like a 10-year-old
by syllogism on Tuesday October 19, 2010 @07:39AM
(#1806793)
As I'd define things, if you're using something for query refinement, then it's part of your IR system. How would you use POS tagging for query refinement though? You can hardly expect to POS tag the user's query, since it's unlikely to be a sentence.
I suspect we have some miscommunication here.
But still, all the obvious ways to use POS tagging for IR have been tried, and don't work. POS is only just marginally useful in machine translation at the moment!
This isn't that surprising when you think about it. What POS tags provide is a small clue about grammatical structure. So you're either setting off down that path and trying to understand a sentence as a tree, or you're going to understand a sentence as a sequence of words. Syntax is hard, and most word-word dependencies are between adjacent words.
There's a local maximum at "just use the words", and POS tags don't take you far enough to get past it for many tasks.