> Let's say that someone infringes on the license. So you take them to court to get them to follow the license. If you have a CLA you are the copyright holder so you can just do this. If you don't, to make your case you have to round up virtually everyone who has ever contributed to your project because they are copyright holders too.

No, if you hold a valid sublicensable license (which includes any F/OSS license, as that's rather the point of Free Software/Open Source Software), and make a derivative work as authorized by that license, you hold the copyright on the derivative work and can enforce it without any participation by the upstream copyright holders.

You may run into difficulty if the offender can show by a preponderance of the evidence that their allegedly-infringing work is derived from the upstream rather than your derivative work, in which case the upstream would have to take action.