if visual scores create problems, why not just obscure them, and use an algorithm to find a better relative popularity between the comments; and show a simple 3 level scale (spam/normal/popular).

If you still need more depth, add a fourth level to the scale.