davetannenbaum

His own work held up very well to replication. It's when he is citing the work of other scholars (in particular, that of social psychologists) that doesn't hold up well to replication.

Yup, completely agree. I think research practices (and incentives for sound research practices) are starting to tilt in the right direction, though perhaps at a slower pace than I'd like.

As someone working in this space, I largely agree with your take.

I'll only add that compared to, say, traditional economics we don't have a sense of how unique the problem is to social psych/organizational behavior. In the past decade psychology has done a lot more replication attempts and data auditing than have other fields in the social sciences. So it could be that other fields are equally problematic, but we don't know.

Most of Kahneman's own work has held up quite well to replication, and that makes up a large portion of the book. But a lot of the social psych findings he discusses have not fared well to replication attempts.

We tested both ideas (altruism and theft aversion) in the paper. Short answer is we find evidence for both. For somewhat technical reasons, we think the "viewing oneself as a thief" component is probably necessary in order to observe higher return rates for wallets with money than wallets without money. But certainly both elements are at play.

It's a fair point, but we've tried to test this issue in a number of different ways. Based on the available data, cross country differences in email usage doesn't seem to have a meaningful impact on our results. See my comment above to gyf304.

A valid concern, but a couple points:

(1) This issue doesn't affect the treatment effect (the difference in return rates for money vs no money), which was the main focus of the paper. If email usage is low in a particular country, that should affect return rates equally in both the money and no money conditions.

(2) We've done a number of robustness checks on the point about email usage and have not been able to find evidence that it has a meaningful impact on the results. For instance, when looking at cross country differences in wallet return rates, the rank order correlation between the "raw" data and one that statistically adjusts for email penetration rates (based on World Bank data) is 0.95.

Actually one of our research assistants was temporarily detained (in Kenya, I believe) for suspicious activity. As you can imagine, having a bunch of "lost" wallets on hand required some explaining.

Definitely a concern. We took steps to examine if there were specific "experimenter effects", that is whether response rates were affected by one research assistant rather than another, and don't find evidence that was a factor. But that doesn't address your broader concern that things might be different if the person dropping off the wallet was native to the country. FWIW the wallets were designed to signal that the owner was probably a local and not a foreigner.

One of the study authors here.

On targeting front-desk employees, we did this both to allow relatively portable cross-country comparisons and for reasons of internal validity (e.g., if we placed wallets on the ground, then can participants "select" into the study which compromises our ability to draw causal inferences).

On your last point about whether it's dishonest to hold onto a wallet. The question is about the treatment effect --- all things equal, is it more dishonest to hold onto a wallet with money vs no money? We polled nationally representative samples in the US, UK, and Poland and in all three countries most people thought so.

She never taught at Cal State Fullerton. Here is her CV:

https://ucirvine-my.sharepoint.com/:w:/g/personal/eloftus_ad...

My understanding is that there have also been semi-recent replications done in France, Iran, and a partial replication conducted at Santa Clara university. All found the same basic pattern as the original study.

Right --- it's called block randomization.

Ted Chiang is fantastic. Good podcast discussion of this story: https://overcast.fm/+HV3dvbFhA

Definitely a fair criticism. In the paper we test for experimenter effects (are the results different for male vs female research assistants? Are some researcher assistants acting differently in a way that might bias the results?) and do not find any meaningful differences. But this doesn't get to your broader point about the homogeneity of our research assistants (all are undergrads from Western Europe, etc).

My sense is that our results probably serve as a lower bound on reporting rates -- if the person who dropped off the wallet comes across as a local, reporting rates would be higher. But this is pure speculation.

Cost was about US $600K in total.

Absolutely. We thought a lot about this trade-off when designing the study.

The disadvantage of using a clear business-card case over a traditional wallet is clear, in that it is relatively unusual. The advantage of using a clear case, however, is that it affords considerably more experimental control in that you can be relatively certain that every recipient knows what is inside. With a wallet, there will be variation in who decides to inspect the wallet, and that introduces selection effects into the experimental design (i.e., are those who are willing to look inside a wallet, compared to those who don't, different in their degree of honesty?). This makes interpreting the evidence a lot more challenging.

FWIW we examined how our measure of civic honesty compares to other known proxies of honest behavior (tax evasion, corruption, etc) within and between countries. If there was something artificial or unique about our setting -- such as using unusual clear business card cases -- then you wouldn't expect our results to generalize or correlate with other measures of honest behavior. However, we find response rates correlate very highly with these other proxies of honesty, suggesting that they are tapping into some broader construct.

Thanks for your comments.

1. This is a fair point. In the Supplemental Material, we explore cross-country differences in email usage. When we statistically adjust for country-level differences in email usage (using World Bank data), the country ranking remains essentially the same (adjusted rankings correlate over 0.90 with non-adjusted rankings). Also, when you restrict the data only to drop-offs performed at hotels -- which tend to rely on email more than other settings -- you see the same pattern of results.

2. Also a good point. However, there are mechanical problems with using the marginal differences between conditions -- for example, countries with high reporting rates in the NoMoney condition will be naturally capped in the possible size of the treatment treatment effect, compared to those with low reporting rates. Because the scale is bounded at 0 and 100% you're also fighting against reversion to the mean at the low and high ends of the distribution. FWIW we find that absolute levels of reporting rates correlate very highly with other known proxies of honesty both within and between countries (measures like tax evasion, corruption, etc), whereas relative differences between conditions do not.

3. We explicitly test this by randomly varying whether the wallets contained a key or not (valuable to the owner but not the recipient), while holding the rest of the contents in the wallet constant.

Many thanks.

Yes, we only went back for those who contacted us first. Your idea is an interesting one, but it gets at something else (how many people, who otherwise would keep the wallet, instead would turn over the wallet when confronted by the owner). We collected wallets from those who contacted us to rule out the possibility that they are returning the wallets empty.

We originally planned to include Japan but after some initial pilot testing we realized that the country was unsuitable for methodological reasons. Japan has a lot of small “police booths” where people can return lost objects. During our pilot tests, we found that Japanese citizens would not contact the owner but instead drop them off at a nearby police booth. This feature made it virtually impossible for us to assign individual wallets to particular drop-off locations.

Quick clarification: we did adjust for purchasing power parity across countries (but not within countries). As you surmised, even for differences between cities like SF and Fresno, the PPP adjustments would be negligible for the wallet amounts we were using.

Thanks for your comments (I'm one of the authors on this paper). For a subset of the countries, we went back to retrieve the wallets (logistically, collecting the wallets turned out to be extremely difficult, which is why we didn't collect them everywhere). We find that over 98% of the money was returned, so doesn't seem to be the case that people are contacting the owner but pocketing the cash.

Interesting how their review omits the paper that has conducted the most direct test of loss aversion:

https://pubsonline.informs.org/doi/abs/10.1287/mnsc.1070.071...

Also they claim the body of evidence doesn’t find support for loss aversion. But a meta analysis on the topic does in fact find support for loss aversion (albeit the magnitude is probably smaller than originally thought):

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3189088

The reason why it is usually considered irrational/illogical is because hyperbolic discounting leads to dynamically inconsistent preferences. For example, under hyperbolic discounting I may prefer $100 today to $150 in one month, but prefer $150 in 61 months to $100 in 60 months. It's the same rate of discounting in both cases ($50 for an additional month of delay) but my preferences are not consistent.

Of course, there are other criteria for rationality besides coherence/consistency (as the wikipedia article alludes to).