I'd say it's mostly a signal to noise ratio issue.

Google famously said they didn't really observe correlation between alma mater and performance BUT it only applied to folks they actually hired.