Yea I agree. It all relies on (2) but they do a very poor job explaining what exactly the simulation is. If there is a paper by McNight, it isn't cited and I can't find it. Can't find it on his webpage[0][1] or google scholar.

This is a non-trivial simulation so the details are really important! Shame they are nowhere to be found.

[0]: https://ibi.gmu.edu/faculty-directory/patrick-mcknight/ [1]:https://scholar.google.com/citations?hl=en&user=sH44LC4AAAAJ...

The article gives us a few clues:

1. the DK-effect supposedly increases the noisier subjects estimates of their abilities are, 2. the graph relies on breaking the data into quartiles 3. experts are less noisy in their estimates of their abilities than are non-experts.

There is also the paper: https://scholarcommons.usf.edu/cgi/viewcontent.cgi?article=1...

Still I think the article does a disservice in not laying out the argument in plain language. I don't particularly feel like puzzling it out

Exactly! Why does the author not tell us how the McKnight model works?

This is the only substantial claim the author makes, yet the result is delivered in two measly sentences.

I feel snookered.

The other paper by Ackermann also has a simulated graph, but I have yet to find the simulation model.

Point 2 also had me scratching my head. The supposed evidence for it (that graph) is statistical mystery meat.

If the two metrics (confidence and ability) were independent in a simulation, you'd expect to see a more-or-less flat line on confidence if you sorted by ability. So the model clearly insists on some sort of association between them, no? If the association is weak, you get a cross-over. Big deal.

Also, the graphs don't really look all that similar - especially when you consider the fact that the line on which the associated variable have been sorted by quantile is always going to be ascending (duh).

Tufte says "to clarify, add data" - one thing that neither the chart from the original study nor the simulated one respects. Surely we'd learn more from also being able to eyeball a scatterplot.

As I understand it, the original study suggested a bias, which causes us to assume that our abilities are closer to the mean distribution than they actually are. The example about the language test misses out on this entirely.

(Regarding that example, a person may even withstand the confrontation with the test score, like, "I guess, even a native speaker doesn't get more than 65% on average", or, "99% is surely not that unusual, everybody with a decent understanding should get a score like this".)

On top of that, the graphs are only barely similar, and I would not call them the same. There is a clear difference in magnitude.