You get one or two huge winners out of hundreds, and they are so huge that they are bigger than all the others combined. The investors care most about getting these rare winners, and if YC does any kind of controlled study on who becomes these winners, I don't believe there will be any statistical significance to speak of.
Since the winners are what they are after they can omit the interview process.
I have a feeling that eventually there will be a Vangaurd of startups that will massively outperform traditional VC/incubators due to reduced management fees similar to index funds vs. regular funds.
It's well-known that the most successful startup in a group of startups is likely to earn more in profit than all the rest combined. We're here on HN talking about PG because YC has been the most successful of all incubators. However, there's always going to be one incubator that significantly outperforms all the others as a result of the distribution of startup success.
I don't know enough about how well PG's investments have done to make informed judgments, but only a very select few people are likely to have that info. I suspect those people have also done their due diligence to figure out if YC is a value-add compared to other incubators (it should be possible to do this without a convoluted double blind experiment), but anyone who's not them really can't say for sure that that's the case.
The key is to perform proper Bayesian updates in the face of evidence in either direction; if you do, as long as your prior wasn't totally insane it doesn't really matter where you started.
So PG's claim could very well be supported by proprietary information.
Never been accepted to YC, but I gather that the value of access to YC's demo day and a strong network of investors and alumni is far from trivial.
So the question with luck is how far down the causal ladder you start considering things inputs (i.e. things which may or may not influence the outcome) instead of outputs (i.e. things which might be the result of those inputs or of luck). So with YC, is that getting or not getting into YC is an input variable that we could check against success, or is getting a YC spot one of those proximal factors, and the luck was just in getting that, at least in part?
So the "is it pure luck?" question isn't very well defined, but the GP's argument might still be useful in answering it. Moreover--larger questions of luck aside--it would be helpful in determining whether the YC selection process is actually useful. But probably only if they didn't tell anyone about it.
For full double-blindness you could even do interviews with the EXPERIMENT group, then use a program to automatically either use the interview result or not. This might be aggravating for reviewers, though, seeing their work wasted by random choices.