News for hackers. Stuff that matters.
Re:How To Build a Naive Bayes Classifier by juanre on Monday February 27, 2012 @03:20AM (#3638205)
Nice article, but I suspect the implementation will not work. I did essentially the same for an AI-class exercise, and was thrilled to see that you could write a working Bayes classifier in 60 short lines of Python code. But later I landed a free-lance job that required writing a classifier that could be applied to real world data, and I soon realized that repeated multiplication of numbers between 0 and 1 sends you to zero too fast for the implementation to actually work. I might have missed it in the code, but I think he's doing the same mistake: you need to normalize or move to logarithms for the estimation of probabilities to work for medium or large datasets.
Re:How To Build a Naive Bayes Classifier by bad_user on Monday February 27, 2012 @04:39AM (#3638383)
I'm the author of the article ...

Yes you are right, it's better to convert the whole thing to a sum of logs, otherwise you end up with floating-point underflow.

The article was getting already too long however, but I'll add a note about it because it is an important optimization that affects both speed and even correctness (because of the underlying limitations of floating point).

UPDATE: note added.

Re:How To Build a Naive Bayes Classifier by ppod on Monday February 27, 2012 @08:26AM (#3638855)
Is the sum of logs method mathematically equivalent to the multiplication of probabilities (i.e. will it always produce a monotonic ordering of class predictions)?
Re:How To Build a Naive Bayes Classifier by alexchamberlain on Monday February 27, 2012 @11:04AM (#3639397)
log(AB) = log(A) + log(B) for all A, B real numbers
Re:How To Build a Naive Bayes Classifier by phaedon on Monday February 27, 2012 @11:20AM (#3639446)
Only for A, B > 0, but that's good enough for probabilities (except 0, I guess)
Re:How To Build a Naive Bayes Classifier by alexchamberlain on Monday February 27, 2012 @11:34AM (#3639501)
That was there to test you...
Re:How To Build a Naive Bayes Classifier by moultano on Monday February 27, 2012 @03:39AM (#3638242)
Even better, convert everything to log(p/(1-p)) log-odds, and halve the number of things to sum.
Re:How To Build a Naive Bayes Classifier by webspiderus on Monday February 27, 2012 @03:32AM (#3638231)
I think this should also have the added effect of being overall faster, as doing lots of addition would be quicker than doing lots of multiplication (log notwithstanding).
Re:How To Build a Naive Bayes Classifier by Radim on Monday February 27, 2012 @05:51AM (#3638524)
That could be true in a hardcore numerical computation, assuming you know enough about your architecture and compiler to make use of it.

In the context of this article and implementation, the speed difference between + and * is utterly irrelevant and unmeasurable.

Re:How To Build a Naive Bayes Classifier by eduardoflores on Monday February 27, 2012 @08:51AM (#3638934)
On Ruby you can use the Decimal class, although it's really slow