Defining the number of items in a set requires the existence of natural numbers in the first place. (In typical set theory, people start with the existence of sets and then define natural numbers from sets.) And it doesn't help when dealing with sets that are as numerous as the natural numbers, or more.

That said it's not wrong to lump together all infinite sets and say their size is infinite. That's how third graders understand the size of a set anyways. It just isn't precise.