Here's how I think about our methodology. Car seat laws generate an unusually sharp prediction for births that can be distinguished from a lot of competing effects (not perfectly, of course, but compared with a lot of other demographic effects)
We're not testing whether the state and years with more stringent car seat laws have lower birth rates. That could be due to tons of stuff going on in those states.
We're not even testing whether the states and years with more stringent car seat laws have lower birth rates just for third child births. That could be relatively fewer things, but still a lot.
Rather, we're testing whether states and years with more stringent car seat laws have lower third child birth rates specifically for families whose first two children are below the mandate for their state and year. And this mandate is itself changing over time for both states and years, so we're talking about different ages of children in question depending on which state and year it is.
This clearly doesn't rule out everything. But the set of candidate explanations for what else could be going on for those specific families in those years is a lot smaller than the broad demographic effects we started with.
How deeply do you understand these very clear confounding variables?
Next, we also add income by year and income by county fixed effects. So if richer people in a given county have more children relative to the average of all two child families in that county and year, we'll also control for that. If richer people in a given year have more children compared to the average of all two child families in that county and year, we'll capture that too.
The point isn't that you need to add every one of these complicated fixed effects to find our result. It shows up in simpler tests too. It's just that you can control for a lot of potential confounding effects, and it still seems to be there.
This is true, but it's true for almost every observational study, and IMO too easy a dismission. You can use this reasoning to dismiss, or more likely, to cherry-pick probably 95% of all scientific results from economic/medical/social/nutricion/climate science. Just imagine the actual mechanics of setting up a proper Randomized Controlled Trial of seat belt safety!
Back to the actual study, the reasoning appearch to hinge on the variation of outcomes between 1) having a 2nd and 3rd child 2) between car and non-car owning families 3) accross different states in the US with different introduction dates between car seat laws. Of course, their might still be lurking confounders, but the presumption that this is really caused by car seat laws sounds quite convincing to me[1]. I'd say the onus on sceptics in this case is to actually propose some possible confounders that might explain variation in all of 1), 2) and 3).
[1] Disclaimer: I haven't gone through the details of the paper to see if the idea is actually well executed. This paper is likely headed for a serious economic journal, where it typically goes through at least days of labor to scrutinise the execution/details/methodology in peer review. I'm not invested enough to volunteer for that.
> This is true, but it's true for almost every observational study, and IMO too easy a dismission.
It's also a correct dismissal. The vast majority of results don't replicate. It's super-easy to come up with results like this paper just by playing around with what you control for (with a 5% of a false positive each time) until you get the result you want. It's almost impossible to tell if this happened. There are many similar issues.
> You can use this reasoning to dismiss, or more likely, to cherry-pick probably 95% of all scientific results from economic/medical/social/nutricion/climate science.
And you should! You should believe results only once they've been replicated multiple times, by multiple communities, with multiple methodologies.
> I'd say the onus on sceptics in this case
The onus is on the people presenting the result, in all cases. That's the scientific process. The authors need to defend the null hypothesis as vigorously as you can.
> This paper is likely headed for a serious economic journal, where it typically goes through at least days of labor to scrutinise the execution/details/methodology in peer review. I'm not invested enough to volunteer for that.
Have you ever gone through a peer review process? Seriously?
I understand you're not invested enough to do that, but neither are the reviewers. It's a volunteer position with no credit or upside. Most give a cursory skim, and either accept or reject. It's no better than chance.
Seriously. NIPS did a study, and found it is literally no better than chance.
Of my own papers, I've only had one where peer reviewers did an in-depth read. For the rest, acceptance was completely random. Reviewer comments often have little relation to the actual paper (many reviewers just skim the abstract and the intro, and don't even fully read those).
The authors posit plausible evidence (observed variance across states with different introduction dates of laws, car ownership, going from 2 to 3 children) that car seat laws was a deterrent to having more than 2 children. A substantial criticism would be to come up with some possible confounders or alternative model of the variation. Saying that it might be caused by anything is a low effort dismissal.
> Have you ever gone through a peer review process? Seriously?
Yes I have, on both sides. Maybe it differs by field, but I (and my fellow PhD students) took it serious enough that we rarely read and finished writing a peer review within a single day. (This was mostly on economic theory / methodology papers, where even understanding the paper can take multiple days.)
I am dismissing it as:
One paper = interesting hypothesis
Multiple replicated papers = theory
Multiple replicated papers, with substantially different methodologies = scientific fact
> Yes I have, on both sides. Maybe it differs by field, but I (and my fellow PhD students) took it serious enough that we rarely read and finished writing a peer review within a single day. (This was mostly on economic theory / methodology papers, where even understanding the paper can take multiple days.)
Interesting. I've worked in several fields, and if this is the case, economics is pretty special. Good on you!
It wasn't always this way. As an economist, you study incentives. Think about how to prevent your field from sliding the way others did. Right now, academic incentive structures dictate cursory skim, and cultures+behaviors tend to (often after many years) align to incentives. As someone in a field (apparently) further along that path than economics, all I can suggest is to act sooner rather than later. It's easier to preserve cultures of integrity than to fix cultures of the opposite.
Lecture me again about incentives :)
This conversation illustrates the challenges with the reproducibility crisis. There is a fundamental problem in how we do science and that problem is hidden by our everyday experience of how effective science has been. This experience colors our perceptions of almost anything that gets dressed up as serious science, including objectively serious science. The scientific process is idealized as this effective tool but research happens at the behest of imperfect institutions and fallible people. Institutions and people are driven by financial and reputational imperative as well as the ideals of their research goals. These can cut strongly against the interests of performing meaningful research and publishing good science.
Having skeptical priors about research in general, whether incentives, error rates, or whatnot, is totally fair.
I look at the first paper as generating a hypothesis. That's the first step in any scientific process. A few replications later, we've got a theory, and then a few more later, a fact. The first papers are as important as the rest of that pipeline.
What causes outright harm is:
1) Papers with poor methodology;
2) Authors overselling their results; or just as often
3) Readers / reporters giving too much confidence in preliminary results
To be clear, I was calling em500 on #3, not the authors on #1 or #2.
On a skim, I don't see obvious problems (but, then, I wouldn't expect to find them; most require a deep read). It's published in an open access venue, which is always a good sign. It also helps that you're from BC, which for the most part, (still) has a pretty good reputation for scientific integrity (there's been a rapid decline in scientific integrity over the past two decades, and that's hit the institutional culture across the river from you particularly hard).
So you're okay by me!
Most researchers are incentived to churn out as many papers as possible. Most people who are making research grant decisions have little incentive to give grants to substantial science over insubstantial science that merely looks good. After all they're unlikely to face career consequences for funding science which doesn't replicate. Many of the people involved in the production of science are disincentivezed to point out problems with the process of science because science funding depends on science being highly esteemed, which only indirectly and with significant time lag depends on the effectiveness of science.
All this leads me to the conclusion that, yes, it is entirely possible that 95% of science in those fields (and maybe others) is junk.
I'm don't think that's farfetched at all. The problem though is that people will not apply uniform 95% skepticism to all studies. They'll just embrace or dismiss them based how how aligned the results are with preconceptions.
> Intervention: Jumping from an aircraft (airplane or helicopter) with a parachute versus an empty backpack (unblinded).
In realty I think a proper trial is supposed to always use the best known treatment as control.
Especially when you consider that something like a state law mandating car seats is only gonna exert upward pressure on the cost of a kid for people with higher incomes. Basically the poors are gonna pile into whatever vehicle they have and the law can fuck off. Nobody says "gee we shouldn't have another kid because we'll need X, Y and Z' (one of those being a bigger car in this case). That's just not how people make decisions at that level. If you want a kid you get a kid and you roll with it. You know it's gonna be expensive but you just deal.
Controlling for income is really really hard since basically everything about lifestyle has some relation to income. Controlling for income with sufficient accuracy that you can tease out an effect that itself correlates with income is gonna be neigh on impossible when you're looking for something that's a 2nd order effect. This is like trying to predict trout populations in Canada by chemically analyzing reindeer piss in Russia. You can do it but it's an exercise in measurement and it's gonna have zero practical utility since sooooo many other things will have affects that dwarf those of what you're measuring.
I think it's far more likely that rear seats have shrunk and car seats have grown because fertility rates have declined and regulators and manufacturers are therefore emboldened to do things that would be laughable in a world where three kid households were more common.
The way I think about it as that these laws only need to affect some families that are on the margin. Are there some families that are just on the fence and tossing up whether to have a third child? It seems likely to me. As long as this cost is a significant one, it just needs to move the needle for these groups to show up in aggregate effects. Lots of people will make the same choice either way, whether it's to have a third child or not have a third child. Our predictions are for people right on the margin.
>Controlling for income is really really hard since basically everything about lifestyle has some relation to income.
We use county-by-year fixed effects to take out the average effect of birth rates in that county and year, whatever form that variation takes. So if it's just that Cook County in Illinois got richer one year and so had more children, this will be absorbed by the fixed effects, and our car seat impacts are considered relative to this baseline. We also control for birth rates for that number of children and age combination, so we're comparing essentially two families with a six year old and a three year old, but in one state/year they both have to be in car seats, and in the other one only one child has to.
We fit three small kids in a 2004 Focus Wagon, but it was an exercise in finding the exact seats that would fit.
It has a lesser crash rating than modern cars, but the reason is there are structural elements that drivers legs could hit in a crash (https://www.euroncap.com/en/results/honda/cr-v/15568). Basically I thing the car is too stiff for Euroncap, but it does not make it less safe in my opinion. Crumple zones are sized for standardized impact tests.
Acoustic stuffing probably takes much space two.
And I agree for the pillars.
Airbags mostly take up what on other cars was dead space behind trim.
Crumple zones #1 function is to buy time for the airbags to deploy between when the front of the vehicle starts slowing in a crash and when the cabin starts slowing. Think about it, if the cabin decelerated at the same rate as the front bumper the occupant would be half way to the steering wheel by the time the bag finished exploding and they'd get hit in the face with an exploding airbag. Crumple zones #2 is deceleration and is tuned for a specific speed of impact. They can be thought of as a single use coil spring without a damper. Too slow of an impact and they do little to nothing (no big deal because it's a slow impact). Too fast and you will blow right through your available travel and bottom out hard. Because the car manufacturers are not staffed by idiots they respond to incentives and tune their crumple zones to be most effective at crash absorption at the high end of the speeds they are tested at.
Most of the increase in "wall thickness" so to speak, of vehicles is from pillars and body structure that increases in width (to get more strength without adding too much material and weight). Then the trim is layered over that and that means you not only lose the space where the reinforcements are but over the whole vehicle.
You see this with the ridiculously high belt lines that are "for safety" but have the convenient side effect of reducing the amount of glass.
No. They really don't. You can shove airbags in anywhere and the passenger cabin is not a crumple zone and is supposed to be rigid. The reduction in interior space comes from fatter structure in the pillars and the various other reinforcements (for that cabin rigidity). Then you add fatter interior trim (to hide fat wiring harnesses and reinforcements and thick sound deadening while still providing clean-ish lines) and the effect compounds.
(I buy the hypothesis. Just too much fun to not make that joke.)
There's an implicit assumption in your comment that the methodology is good, and that anyone who criticises it must be unable to see that. Have you ever read the story "The Emperor's New Clothes"?
"It wasn't explained to me, therefore it is invalid" is not valid.
And the highschool student would be right, and the expert would be wrong.
It doesn't matter how much of an expert you are if your conclusions are wrong because you don't know statistics.
If anything, an "expert" like that is actually a complete fraud.
Additionally, interpreting the results as t-statistics doesn't really tell us very much, as the variation between model and reality could easily be noise from the control variables rather than from a relationship between car seat laws and fertility.
I'm not saying the paper doesn't have any merit at all--I think the idea is quite interesting. I am just skeptical of inferring anything from the results, given that the methodology is very poorly described (and is so error-prone).