I would totally agree, and this is why I use a different password for every site.

However, I am in the small minority and we have to face the fact that most people use one or only a handful of passwords, and these people have a very poor idea of the criteria for a good password, mainly because so many sites have poor guidelines, and poor policies (like max 16 chars, no punctuation, etc).