Only in locally-small categories. But yes, category theory often makes use of sets.