That expression clearly implies that MOST immigrants are drug dealers, criminals, or rapists - and that he can only assume that a few of them must also be good people (he can't know for sure that any are, though, he can only assume it - implying he has only seen the drug dealers, criminals, and rapists).