If you are aiming for elite (or semi-elite) schools you need sports or clubs to appear well rounded. 95% of students aren't really trying to get into those colleges.

A lot of public colleges essentially create a formula (GPA and SAT/ACT) and the "soft factors" likes clubs and sports only decide when you are in the gray area.