I think this change would be for the better. What about something even easier to see, than +2 or -2 and hovering with the mouse? Most people aren't going to mouse-hover ... And it can be good to know if something has 103 upvotes and 100 downvotes, because then it's probably a rather dangerous comment/advice.

Maybe showing the text "Some disagree with this", if some people disagree? Or "Many people", if many. I linked to a demo about how that can look, in another comment here — search for "Look, a demo" on this page.