I think it's obvious from the context that he means "worst" from a macro perspective. I don't really see why it bugs you so much -- the fact that any given person likely prefer a "kind" slaveowner, or prefer that the rich be charitable seems to go without saying.