Re:Comment spam random text template
by tikhonj on Monday April 22, 2013 @02:32PM
(#5590963)
Sure, there can be millions of unique texts, but they all follow certain very similar patterns. That's what would make even a simple naive Bayes filter very effective against them.
[ Parent
| on HN ]
I think you're misunderstanding how naïve Bayesian spam filters work. These techniques explicitly do not recognize patterns at all, they just look at vocabulary, and treat each word as a completely independent feature—so they don't even recognize "discount viagra" as a pattern, but merely two independent features "discount" and "viagra".
First of all, you can have any sort of features you want. So you could certainly look at things like sentence structure, pairs of words and so on.
More importantly, I used "pattern" in a very general sense: there is strictly more structure to a post generated with this system than a random post or even a human-authored one. So repeated words between posts are a pattern, and a very important one at that.