Ask HN: Do you have any code or projects of your own you would like to talk about instead?

Re:Ask HN: Do you have any code or projects of your own you ... by Zak on Wednesday April 29, 2009 @11:50PM (#586355)

I'm working on implementing a hyperspatial text classifier in Haskell. It's conceptually similar to the hyperspace classifier in CRM114 (essentially a weighted nearest-neighbor search), but shouldn't crash as much or corrupt its data files. I've thought of some interesting problems to point a fast and effective classifier at that aren't spam detection.

I'm also working on a library/framework in Clojure for apps that process a folder full of files in some way. The idea grew out of a Clojure program I wrote to make a folder full of images greyscale (http://github.com/zakwilson/imagesieve/tree/master).

[ Parent | on HN ]

Re:Ask HN: Do you have any code or projects of your own you ... by mahmud on Thursday April 30, 2009 @12:46AM (#586403)

Zak, by classifier I hope you mean "categorizer". i.e. given a text and a list of categories, return which categories the text belongs to, along with a confidence number.

If so, I say chase that thought a bit. You might save me from actually messing with NLP.

[ Parent | on HN ]

Re:Ask HN: Do you have any code or projects of your own you ... by Zak on Thursday April 30, 2009 @01:38AM (#586453)

Yes, I do. The usage is essentially identical to the Bayesian probability classifiers commonly used for spam detection. I compared several of the classifiers available in CRM114 and found hyperspace to be the most accurate as well as being much faster when checking a large number of categories.

[ Parent | on HN ]

Re:Ask HN: Do you have any code or projects of your own you ... by mahmud on Thursday April 30, 2009 @02:12AM (#586473)

Amazing. This is pretty cool :-)

The thing is 100% statistical right? It doesn't know anything about the grammar of a natural language (say, English.) right?

[ Parent | on HN ]

Re:Ask HN: Do you have any code or projects of your own you ... by Zak on Thursday April 30, 2009 @02:27AM (#586485)

It's entirely non-statistical. It works by measuring the Hamming distance between samples using features consisting of single tokens, sequences of tokens and sequences of tokens with gaps. For example:

The

The quick

The quick brown

The quick ... jumped

would all be features of the input "The quick brown fox jumped over the lazy dog".

It doesn't know anything about grammar, though as a performance optimization it might be useful to limit the length of features to detectable sentence boundaries. I intend to make that behavior optional. More information here: http://crm114.sourceforge.net/docs/KNN_Hyperspace_Filters/KN...

[ Parent | on HN ]