News for hackers. Stuff that matters.
Re:Ask YC: What new technologies are you exploring? by lsb on Monday November 19, 2007 @06:25PM (#81717)
Large-scale text mining (Google/Wikipedia). There's boatloads of patterns and data latent in an enormous popular text corpus.

For instance, if you have two terms, and you want to see how similar they are, that's P(A|B). So do a search query for A&B and for B, and P(A|B)=P(A&B)/P(B), and that's a first approximation, a Google Distance, for arbitrary terms.

The information silos are only a couple years old, and already they're so valuable. I can't wait until the Internet's a century old.

Re:Ask YC: What new technologies are you exploring? by pixcavator on Monday November 19, 2007 @07:56PM (#81754)
P(A|B) != P(B|A), so not really a distance...
Re:Ask YC: What new technologies are you exploring? by lsb on Tuesday November 20, 2007 @05:15AM (#81888)
It's from Cilibrasi and Vitanyi. They have shimming factors, so you take the min of A|B and B|A, then you throw in some logs to not penalize low probabilities.
Re:Ask YC: What new technologies are you exploring? by aswanson on Wednesday November 21, 2007 @07:42AM (#82428)
He's onto something here. What tools/languages are you using for this.
Re:Ask YC: What new technologies are you exploring? by lsb on Friday November 23, 2007 @05:17PM (#83067)
When I'd stopped independent work on it, and brought it to Songkick, it was coded entirely in Javascript, with a reimplementation in Ruby (to check work). Email me (leebutterman/gmail) if you're interested.