The point is, you don't know they legitimately wanted a ladder, certainly not from what the resident knew at the time. All they see is someone who seemed out of place and who may have exhibited the suspicious behavior I described above.
If you've never been in a neighborhood with high crime, or where scam artists have preyed on the naive, great, but don't act judgemental toward those who may have.
I find it hard to believe that anyone goes about their daily lives with that level of paranoia towards every new person they meet. I think it's much more likely that it's because he's black. . . especially considering the background of negative race relations and stereotypes about black people and crime.