Thank you for putting into clear words my gut impression that this wouldn't produce the intended result.

I wonder if you couldn't do better with binary questions. I'm reminded of the site http://www.correlated.org/.

Suppose you were to ask a bunch of binary questions, and then use some statistics to find questions that have fairly evenly distributed and non-correlated results, then take 24 of the best questions by those criteria (arbitrary, but 2^24 = 8^8) and find matches.

So basically, ask a whole bunch of binary questions, find ones with roughly 50-50 answers that don't correlate with each other, make the match-ups based on those and ignore the others.

okcupid does exactly this. Most of their questions are submitted by the users themselves, and they have hundreds if not thousands of them, but they present the questions to you in the order of most differentiating first. You can answer as many or as few as you want, but matches get better as you answer more, with monotonically diminishing returns. You also get to say which answers you would accept from your match (doesn't have to include your answer) and how much it matters if at all.

My gut tells me it's akin to doing principle component analysis. To find the most differentiating question, do 1 dimensional PCA, and the question most aligned with the resulting dimension is question #1. Then do 2 dimensional PCA fixing the first dimension as the first question. The result is question #2. And so on.

OF FAR GREATER CONCERN: They are collecting email addresses associated with personality questions. There is no privacy policy, other than the statement "If there's a match, we'll email you. (You will not be emailed under any other circumstances: no promos, no newsletters, nothing except news of a match.)" You don't know to whom you are giving this information, as there is no named legal entity, just a gmail address and a domain name whose registered owner is hidden in whois databases.

I suspect OKC can make much more accurate predictions than this site, as they have vastly more data (some users answer thousands of questions, most users who are active for more than a few months have answered a couple hundred), vastly more users to compare, and they are pretty savvy about statistical analysis (reading their old blog posts is quite educational on the subject).

But, even given all that data...I find about half of my 99% matches (of which I've talked to maybe a half dozen and gone on a date with maybe four) aren't particularly compatible or interesting, to me. There seem to be other factors at play that just aren't measurable. And, heck, maybe being just like someone else isn't what makes an ideal friend/lover/partner.

It's like a Gram-Schmidt process. You take one question, then you look at other questions given (quotiented by, projected on, etc) that question, and so on.

> My gut tells me it's akin to doing principle component analysis

Or a decision tree, maybe

Since I can't edit, here are some more details on what OkCupid does http://blog.okcupid.com/index.php/page/3/

or like Akinator: http://en.akinator.com/

This site is amazing.

It could be argued that certain types of people are more likely to exist and thus more are matched.

EG: The very first question about what you do at a party.

The "sit on the couch and observe" type of person is probably one of the more rare options than the "center of attention" or "dancing with people" or whatever other options there were (I forget the options I didn't pick myself)

I'm not a party goer and there wasn't an option to not attend the party. So I'd be on the couch watching other people. Not particularly a popular thing to do at parties.

Even then, it’s all the more important to find the questions with the highest entropy, given that you know someone’s already in a popular group. It’s all about evenly dividing the population based on important factors. Evenly dividing people is easy enough, and finding important factors is easy enough, but doing them both at the same time can be very tricky.

Even distribution is not the goal. Finding like-minded matches is the goal.

From a purely statistical standpoint - their claim is false about the roughly 1 in 16,777,216 chance. But the goal is to find like-minded people.

Let's create a 1 question True/False test. Let's assign the probability of answering True on the question is 70% and answering False is 30%.

You give the test to 100 people. You now have 35 pairings for "True" and 15 pairings for "False". Would it make sense to pair the "True" people with the "False" people to approach an "even distribution"? Only if your goal was to match people with "1 out of every 2 people" from a purely statistical standpoint of weighing 2 options.

The importance in this case is not distribution - but rather if these 8 questions determine who is "similar" in thinking to another person. If you asked someone their favorite color and their favorite pet - you might get a lot of matches. But many of those matches might be terrible with the people having little in common beyond that.

In the FAQ they take a step back and don't really guarantee good matching, even in the event of a match. However matches should be statistically rare (even if biased towards a specific 8 answers on the questions) and might still produce "good results". This makes it an interesting case study and one that can be tweaked and redone if we ever discover a way of asking only a few questions (I'd say no more than 10?) and accurately defining someones personality.

That's the way I see it at least. More of a case study than a statistical claim, even if they're trying to spin the statistics to spur people to take the 8 question quiz. Who knows, it may end up with a lot of good matches based on just-vague-enough questions and the likilihood of similar answers (even if, in some scenarios, fewer than the projected 16~ million and in other scenarios even more)

Even distribution is just important for doing it in as few questions as possible. If you want to do it in 8, and cover 16M distinct personalities, you're gonna need them to be very evenly distributed.

Check out http://en.wikipedia.org/wiki/Entropy_%28information_theory%2...

Information gain is maximized when the distribution is even.

Imagine a very extreme biased form of this where everyone answers the same way to a question, and that question is repeated 8 times with different wordings. You will only get 1 distinct personality type.

For a real-world example, take the myers briggs (MBTI) personality test. It only does 16 distinct personality types and takes about 50q to get right. Only way 8^8 beats that, without any data to predict the distributions and correlations out of the gate, or completely revolutionarily novel psychological theory, is with precogs.

There have been 0 matches with over 12,000 tests. Assuming their claim of 1/1,000~ is anywhere near remotely true - then you may have increased your chances of finding a 'good match' tenfold already with the assumption that matches will be good (there is of course the chance they won't be)

The Myers-Briggs places weight on questions to give one of the 16 personality types made of 8 traits. For example, one answer might give you 80% of INTJ and 20% INTP and by factoring the weights across all the questions will tell you if you were weighted more or less towards INTJ or INTP, even if you were more INTJ for that singular question.

In 8x8 the questions are not weighted in that way, which is an important difference. While there are 16 distinct personality traits in MBTI, there are only 8 traits. Questions can weigh towards those 8 traits in various ways. From my experience MBTI tests range from the 5 "strongly, disagree, neutral, agree, strongly" answer types and the questions are targeted to 2 (sometimes 4, but typically 2) traits.

Meaning if neutral is 50% : 50% (Not leaning towards introvert or extrovert) your answer to the question is effectively meaningless. However if you "agree" it might be 75% : 25% and if you "strongly agree" it could be 100% : 0%. You take the aggravated ratings of all the "I or E" type questions and determine if the user is introverted or extroverted based on which holds more weight across 50 questions.

8x8 does not weight the questions to determine personality traits. The questions must match exactly to determine a match. In comparison, Myers-Briggs can have two people be INTJ who answered questions very differently but their aggregated weights of each question added up to them both being an INTJ.

Again: this should be treated as a case study. Since questions aren't weighted, this is different than most personality tests. This isn't a personality test - but finding someone who thinks exactly as you do in regards to these 8 questions.

The largest flaw is if these are the wrong 8 questions to ask and if matches lean towards "good" or not. Which can be determined after enough matches. Making this a neat case study. :)

E:

TL;DR If 12,000 people took a MBTI I guarantee there would be many matches already. This isn't a personality test - this is a line of thinking test. Whether the questions will result in good matches or not will be left to when there are any matches to begin with.

the MBTI example and use of weighting only makes it more clear that this "case study" is dead from the start, because it shows it takes many questions just to ascertain someone's position on each eigenvector. For example, the most important explanatory personality trait is masculinity vs femininity. How do you ascertain that with one question and 8 answers? For example a question asking if you are male or female only tells you so much and not the whole story. none of the real underlying important explanatory variables can be directly measured with a question, or 8 questions.

And I highly doubt they have no matches in 10k samples, especially given the homogeneity of their audience. And if so then they are just measuring traits of personality that, while diversified, nobody cares about. Check out the birthday paradox. Even if there are 16M possibilities, the chance that two random people in 10,000 matched is much higher than you think.

If it were trying to match people based on their personalities and not their direct line of thinking - you would be correct.

Until the results speak for themselves (ie. accuracy of matches) you have no way of knowing if this is a complete failure or successful venture. Although you are free to speculate that it will be meaningless due to the methodology, you have no evidence to back that up yet.

Many dating sites work on a similar principle, although they all use weighted questions. Weighted questions for personality increase possible matches by narrowing down the preciseness necessary. The goal here is to decrease possible matches by requiring exact matches.

Dating sites want to match you with someone quickly who is "close enough" to be compatible based on a personality test and their user data of what answers pair well with other peoples' answers.

8x8 doesn't care how long it takes to find a match, they want an "exact match" to be compatible based on how you answer arbitrary questions.

8 questions was chosen because to have answers be "100% exact" are already rare. If they chose 50 questions - the chances of an exact match become increasingly rare as to possibly never exist in a persons' life time.

E:

The birthday paradox is 1/365, and I'm well aware of the statistics of someone out of a group of 20 random people sharing a birthday with another person of that same group.

I just did the math and at 12000 people and 16777216 possibilities, 98.63% chance there's at least 1 match. Not buyin' it!

They intend to look for matches offline with a few days delay. The count of people who have taken it looks like it is updated live.

It is quite possible that they have not tried to do the match online. Given their demonstrated competence, I wouldn't be surprise if it comes as a shock to them that they can simply put results in a flat file, sort, and then scan them for matches very quickly...

According to the Reddit page the number of tests and matches found is accurate.

There were 11,300~ tests when the page last loaded for me. They are caching the results of the tests and refreshing non-live time in an attempt to put less stress on the server.

There were 0 matches at that time. While I'm wary of site-counters - I'll give them the benefit of the doubt.

E: The formula is

P(N) = 1 - prod[ 1 - (i-1)/(88 ) , i = 1,2,3,...N]

At 15,000~ there should be a match. So while pretty high at 12,000~ it's still in the realm of possibility that there is not a match, just very unlikely.

There's also the chance they haven't had the ability to check for matches because their servers been down so often so they haven't been calculating for them. So there is a chance that there is a match in the data but it hasn't been calculated and so wasn't showing when I checked at 11,300~

and it only gets worse as possibilities become less evenly distributed! we are approaching 100% probability this is all bullshit and won't work at all.

for the curious:

    (1 - reduce(operator.mul, map(lambda a: (8 ** 8.0 - a) / 8 ** 8, range(12000)))) * 100.0

> The birthday paradox is 1/365

You mean, for 2 people? It's higher than that http://en.wikipedia.org/wiki/Birthday_problem

>There have been 0 matches with over 12,000 tests.

I really don't believe their number of matches. Why... We're on the Internet, it's full of trolls.

I'm sure any number of people will have taken the test at two different locations and answered the questions the same, just to get a match.

> Even then, it’s all the more important to find the questions with the highest entropy

Also see http://www.gwern.net/Death%20Note%20Anonymity for a discussion of information theory in identifying people.

That was an interesting read - though I skipped after the "game is over part". I'll have to go back and read the rest later for "what he should have done."

As an aside, I'm now worried about serial killers trained in using information theory to prevent themselves from being caught. Although not having a supernatural Death Note makes that a lot more difficult.

Simple stupid example. A lot more than 1/5 of the people would choose A for the following question:

How many people have you killed?

a) 0

b) 1-9

c) 10-99

d) 100-999

e) 1000 or more

Or, for the second problem or correlations:

Do you have a Y chromosome? a) Yes b) No

Do you have two X chromosomes: a) Yes b) No

Each are 50-50 questions, but the answer to A is correlated to the answer to B.

It's hard to find questions whose answers are independent variables.

Well, not quite 50/50... :) https://en.wikipedia.org/wiki/Klinefelter_syndrome

Perhaps not correlated in the way you are think, due to confounding factors:

A) how many people don't understand the questions

B) how many people understand the questions but don't choose to answer "correctly" e.g. transgender. E.g. smart arse Yes, Yes answers.

C) correlation with a third variable e.g. sampling bias (HN readers that answer internet surveys).

The survey instrument could be perfectly constructed and the methodology still grossly flawed.

Because the population only consists of people who are willing to take the test [and have internet access and speak English etc.] generalizing the data is problematic at best.

So what? If you are searching for people who are like you, then they are also likely to be in the same demographic.