Similar example. I'm reviewing a complex hand tool and say about its use-
"you turn it right to tighten but left to tighten, i dunno maybe I don't get it, let's move on"
To some people reading the implication is that the hand tool doesn't work,is wrong, or confusing to use.
If he has mistrust why not attempt to prove one or more of the studies were fake, used inaccurate data, or came to the wrong conclusions based on the data?
Because by simply mentioning the mistrust and bias, AT BEFORE he even discusses the counter arguments, WHILE ONLY providing potential motive as evidence, he manipulates the reader by leaving a bad taste.
Why even mention your mistrust at all? How is that useful in this scholarly article when you don't offer proof? In my opinion manipulation
The more I think about this article the more it just feels like someone so determined to prove his theory instead of using the proper scientific method and being open minded.
Because their research is all about the importance of education. It's like asking whether a pharmaceutical company would strive to publish negative results of their drugs; generally, not if they can avoid it! This is why recent pushes for open data and pre-registration are so important.
> If he has mistrust why not attempt to prove one or more of the studies were fake, used inaccurate data, or came to the wrong conclusions based on the data?
Because that's not the goal of the post. The goal here was to develop a general, informed opinion on a topic by performing a broad review of the state of evidence. Misinterpreting or dismissing one study here or there will not change the conclusion much: if there are strong signals in the data then other studies will surely show it, and if there are no strong signals, then you won't know to trust this one study over the others which found no such signal.
> Because by simply mentioning the mistrust and bias, AT BEFORE he even discusses the counter arguments, WHILE ONLY providing potential motive as evidence
He doesn't provide only motive as evidence, the other studies that dispute the "huge effects" that education researchers seem to find are evidence that these huge effects are not real, hence the suspicion. The whole post to the point where he starts discussing this literature consists of a boatload of evidence that should make you skeptical of those studies.
> Why even mention your mistrust at all? How is that useful in this scholarly article when you don't offer proof?
I'm not sure what sort of "scholarly article" you think this is, or what that term means to you. Hopefully my elaboration above cleared this up. This hand-wringing over one study is just an indication that the evidence is simply not that strong.
I think Scott looked into this with an open mind, as he usually does, but the evidence is simply not that convincing. For low income kids public education is a huge win, probably mostly due to the structure and safe environment it provides, but the evidence for its educational value is just poor.
The author uses home schooling as part of his arguments but doesn't show that either mostly different or mostly the same as being in school for the factors that matter.
I assume what kids are missing isn't being in a specific building, it's probably instructor lead education in certain subjects. However he doesn't discuss what defines school
He does mention "unschooling" which is student controlled and I'll accept that as being different enough from school for his argument.
The problem each parent operates differently and even if there are groups he doesn't show proof that unschooling or the similar is the majority of home schooling
More briefly he makes an assumption that home schooling is different from school in the way that matters when you miss it
From your link, charter schools don't really outperform public schools so I'm not sure that this has meaningful impact on his argument.
> He provides no evidence that a significant amount of parents are operating in a way that is unlike school.
I'm having a hard time understanding how you can think otherwise. In your experience, do students have the same relationship with their teachers and school administrators as they do with their parents? Definitely not in my experience, and I expect in yours as well, so clearly the default assumption must be that there is more diversity in homeschooling than in public or charter schooling.
Another line of argument is that there is clear evidence that smaller class sizes are correlated with better outcomes, possibly because lesson plans are more catered to individual strengths and weaknesses, and there is no smaller class size than one-on-one or one-on-two in a parent-child scenario. Here again, the default assumption is that there is more diversity in home schooling approaches.
> I stopped reading after the intro to section 4 which I will now quote- [...] To be fair , as stated, he does start to look into them. However he had no issues with using studies from researches that worked for his theory but if they work against it then he's skeptical.
He literally explained the reason in the part you quoted: pro-education researchers have incentives to find schooling is essential, but researchers who don't have this incentive therefore don't have have this bias (and researchers who do have this incentive but don't find positive results are very trustworthy). It's the same reason we ask researchers to cite their conficts of interest on every paper, because we all recognize that this does bias results. I don't see a problem here, as long as the skepticism isn't extreme/disproportionate.
> His response to counter studies 2 - Teacher's striking leads to loss learning (US) - He says this in response " don’t entirely understand their methodology, so I can’t critique them very well, but I have to admit I’m skeptical of them anyway" Again he states he is skeptical for no other reason because he doesn't understand the methodology and it goes against him
Per above, the default should be skepticism given the vested interest. Secondly, Scott does this sort of publication analysis a lot, particularly in his practice as a psychologist, so if he doesn't understand a methodology then that's suspicious on its own.
> His response to counter studies 2 Teacher's striking leads to loss learning (Argentina) - he says "I can’t find any specific problems with this study. It just seems extreme to me" he then basically accuses the study of being misleading because he doesn't understand it.
Extreme outliers should immediately raise suspicion. If a signal is that strong then we should see it everywhere, but we don't, ergo we should be suspicious that this was either a statistical fluke or some kind of shenanigans.
This sort of argument is what has led to the retraction of lots of ivermectin papers over the past year and a half. The reasoning is sound. It's not an outright refutation, more a recognition that a single paper is insufficient to accept a conclusion absent corroboration, and we should investigate such strong signals that fail corroboration for fraud.