News for hackers. Stuff that matters.
Re:17 Year Old Builds a Better Search Engine by bambax on Friday July 20, 2012 @05:56AM (#4270111)
This kind of comment is what I come to HN for, so thank you.

I use OpenCalais to compute keywords for texts; where does it stand vs. [6] and LSA in general?

Re:17 Year Old Builds a Better Search Engine by Dn_Ab on Friday July 20, 2012 @07:25AM (#4270267)
At their scale I doubt they are using LSA or anything of similar complexity. My guess would be something like tf-idf coupled with any thing from naive bayes to logic programming for entity recognition.

Final output would likely use some hand curated rules on how to combine the above with infromation from curated databases like dbepida.

________________

There is this thing called random projection based on the observation that in high dimensions, most direction are nearly orthogonal to each other. This is a wonderful observation because it allows us to write stupendously simple (and fast) algorithms that give good results for mathematically sound reasons.

I strongly suggest anyone who is interested in LSA but wants something scalable, simpler to write and faster to look into Random Indexing.

Re:17 Year Old Builds a Better Search Engine by riffraff on Friday July 20, 2012 @06:50AM (#4270210)
on a related note, I've been told once that OpenCalais actually uses a rule-based system, does anyone know if that's true?