That's the most interesting part of this for me - nobody's saying "you're missing some important factors, and let's add them." People are saying, "I don't know that you included all the factors, so I am going to go back to reading opinion columnists for whom 'factor' is not a word."
Has everyone thought about the fact that the alternative to spreadsheets is policymakers putting their hands on their hips and saying, "Yep, $30k sounds about right for this month?"
Ok, so I’ll say it. Caplan doesn’t know what he doesn’t know. And neither did the people who he asked to check his spreadsheets.
What he is trying to do, in an informal hand-waving way, is something known as causal inference. There are formal, robust ways to do causal inference, which Caplan is not using. No amount of checking his arithmetic or adding more factors is going to fix that.
This would be a good place to start: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2943670/
> Has everyone thought about the fact that the alternative to spreadsheets is policymakers putting their hands on their hips and saying, "Yep, $30k sounds about right for this month?"
Well, I can’t speak for everyone else, but I don’t think about that because what you claim as fact is actually a false dilemma. The actual alternative is to do good science using the correct methods.
Why bother with spreadsheets at all then? If the author is going to do a bunch of arithmetic to convince us that some conclusion is correct, shouldn't we ask that he actually perform relevant and useful arithmetic?
1 + 3 = 4 See? I just proved that we should spend eleventy brazillion dollars on K12 education next year.
This is a great point, but has anyone looked at the spreadsheet yet to see for themselves? I think we're proving him right so far, haha.
But we aren't.
The author has described his method in words. My contention is that the method he described does not pass the bar for good science. If my contention is correct, then it simply does not matter what is or is not in his spreadsheets.
Here's an analogy: If I told you that I determined the safest car on the road by measuring the distance to the moon, and that I had very very good spreadsheets for measuring the distance to the moon, would you really bother to check my spreadsheets?
I don't think you would. I think you would (correctly) say, "Hey, that's a really dumb way to figure out the safest car on the road and I'm not going to bother to check your math."
Now, I don't think that the author has done something quite that orthogonal to his stated goal, but I also don't think it's close enough that I want to bother checking his math.
Since the prior is uninformed, it's obvious this isn't "extraordinary claims require extraordinary evidence" since both claims are extraordinary. It’s just that one is political desirous. Ain’t fooling nobody with the attempt at rational cover.
Caplan shouldn't be surprised at the reception of his work, because people have this prior.
Of course as you say high priests of rationality examine their own biases as they reach for the book, and by the time they flip it to read the recommendations on the back they are already empty vessels for unadulterated pure non-chill-filtered high-octane data and nothing else. But most people working in public policy are just lowly humans. :/
Plus it seems despite his many spreadsheets his model is not able to do causal inference, which means his tables are very verbose narrations or appendices to his arguments in prose, and not a decisive proof.
> to change your mind
Unless one is in a state of 100% confirmation bias then ideally every little piece of evidence counts. Sure, we are not perfect walking-talking sentient infinite resolution mathematical distributions, so in practice we simply discard a lot incoming information, as we have meta-(meta?)-heuristics that we depend on. (Eg. if our friends and family and the news and even random blogs claim X we start to take it seriously, but still, if it doesn't really affect us our attention won't bother. And so on.)
But my hobbyist friend lacked what I think is important here. The humility to see that the piles of multiplication for the handwavey guesses that they are. Caplan's spreadsheets, to me, made his argument more persuasive. But still not that persuasive. My priors already have education being a net benefit as highly probable, and I'll update, but not much.
This seems rational to me.
How'd you get those priors?
I'm in no personal position to otherwise say boo.
Basically, Mount and Blade in space but frankly, I think it's a better take on the formula. Excellent modding community (though I'm upset that the Knights Templar and Blackrock mods are still not updated...)
I remember being part of online community for a game (starsector, if you're interested) and a user had taken it upon themselves to create effectiveness measurements for the weapons in the vanilla game and in mods. To do this, they created a spreadsheet that mapped out damage per second, range, energy cost, points to use... ect. Numbered them all together, and ranked them with each other.
Personally, I thought this was a valuable contribution. Plenty of mods go off the deep end with the kit they added to the game, so to me, this seemed like a valuable tool to help keep their kit in line.
But the author? They thought what they had produced was an unassailable statement of fact.
Many people came forward, including me, countering that the actual simulation of the game is substantially more complex than the pile of multiplication and addition they had cobbled together. But like OP's article, those criticisms weren't specifically addressing the formulas themselves, so the author didn't really consider them.
This is a bit of a puerile example of course, I bet Mr. Caplan is substantially better at multiplication than my hobbyist friend trying to categorize weapons in a silly space ship game.