And you are right, I did hire people regardless of their answers to some questions and I would then assess how they turned out to figure out if my questions were any good. I basically gave everyone I interviewed a pass on at least one question even if they bombed it abysmally. Some of those folks turned out to be our best employees so I learned to never hold one bad answer against someone. It also gave me a dataset to improve my questions for the next wave of candidates.
Anyway, I'll attest to the fact that Dan at least didn't veto all candidates purely on the basis of their response to this question (unless I was the result of a massive clerical error).
Outcomes aside, I'm very thankful for my experience at Optimizely. My time there was a major inflection point for me both personally and professionally. I wouldn't be as effective in the workplace as I am today—nor would I have some of my most meaningful friendships—had I not cut my teeth at 631 Howard — thanks Dan (and Pete, if you're reading this too).
EDIT: Also considering the timeline and the role described in the OP's message, there's a nonzero chance we were interviewing for the same role. Small world!
Additionally, I should have considered the possibility that both parties are mis-remembering the interaction. Seems I owe @worldsoup an apology.
@worldsoup I apologize for suggesting you were lying about your experience.
Anyway, if I had to guess as to what really happened, I expect the recruiter told a small, white lie when pressed on why OP didn't get the job.
Probably OP though they'd passed the interview because they got a meeting with the CEO. Or OP thought passing the interview meant they had the job, when in reality there were multiple qualified candidates and Optimizely chose another one.
There's other information in this thread to support this conclusion. I won't rehash it here.
In conclusion: the most charitable explanation is both parties are telling the truth to the best of their memories, and maybe a third party was a bit dishonest in a way meant to spare someone's feelings.
1. The op is lying about his experience.
He states he answered NO to question 2, which is what you claim to be looking for. Unless his answer to question 1 was "Work for Optimizely" (which, I guess, someone might say, maybe) then I don't see how you get a contradiction. By your logic, he would have been a hire.
2. The recruiter lied to him about why he was rejected.
Maybe everyone got a short sit down with the CEO back then and he took that to mean he passed all the other interviews? Maybe something else didn't check out and that was an easy way to let him down?
Anyway, if you're A/B testing, wouldn't you hire people regardless of their answers and then assess their performance over a longer timeframe to determine the efficacy of the questions and answers?