Agreed. I think OP was more referring to crime. Ok maybe Chelsea vs Croydon is a bit different, but compared to any of the surrounding areas e.g. Wimbledon and Clapham, it isn't much different.
(I was in South Wimbledon - close to Mitcham and Tooting - for 2 years)
Most of the worst boroughs in London in terms of crime are the inner-city boroughs, regardless of wealth (Westminster in particular is a crime ridden hellhole if you look at the raw numbers by population, due largely to the amount of business and nightlife and low number of people living there; but Croydon city centre suffers from some of the same effect, dragging the overall crime rate for the borough up, even though the totals for the borough still remain better than average for London).
Many people love it however.