Re:Are Experts Real?
by eesmith on Monday January 11, 2021 @03:13PM
(#25735803)
Teller attributes Bohr as saying: "An expert is a person who has found out by his own painful experience all the mistakes that one can make in a very narrow field."
I prefer something closer to "An expert is someone who knows some of the worst mistakes that can be made in his subject, and how to avoid them". 'Cause even experts make mistakes in their own field.
This piece seems to think an expert cannot make a mistake in a given field of expertise, which is "utter nonsense, of course."
I don't like this example: "If you look at the papers citing Viechtbauer et al., you will find dozens of them simply using N=59, regardless of the problem they're studying"
Paging through the linked-to Google Scholar search finds 90 results. Of the first 12 with readable text, only two seem correctly described that way (and max 3), suggesting under two dozen examples. A spot check of the remainder suggests that pattern holds. I'll show my work:
#1 says "A sample size of 30 patients per group was chosen as recommended for pilot studies to achieve an appropriate level of statistical power [30]." They selected 60, and "1 patient was lost to follow-up" giving 59. So this wasn't N=59 but N=60. - https://medinform.jmir.org/2019/4/e14044/?utm_source=TrendMD... . 30 was chosen because "It corresponds to the detection of a potential difference of 21% between groups for a power of 80% and an alpha significance level of 5%." which is not the default example from Viechtbauer.
#4 specifically does NOT use Viechtbaueret, writing: "Viechtbaueret al., (2020) recommends sample size between 10-40 as adequate for feasibility study. Linacre, (2015) suggests a sample of thirty (30) respondents as appropriate for general purposes. This study however uses 60 respondents becauseadequate sample and item produce a good item, person separation and reliability in Rasch measurement (Fisher, Elbaum, & Coulter, 2010)." - https://journals.sagepub.com/doi/pdf/10.1177/0306624X1985848...
#5 uses N=43 ("With an alpha value of 0.05 and with an 80% power we need a sample size of at least 43 patients to detect a difference in validity and assess reliability") - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6886622/ . This is not the default example from Viechtbauer.
#6 uses N=59, using calculations directly from Viechtbaueret. It appears to be used correctly.
#7 uses N=53, commenting "There is no gold standard for sample size calculation in pilot studies. Sample sizes as small as 10 (Hertzog, 2008) and as large as 59 (Viechtbauer et al., 2015) have been recommended." - https://journals.lww.com/nursingresearchonline/FullText/2018...
#8 uses N=30 for each region, writing "Tamaño de la muestra: en cada región, se llevó a cabo un estudio piloto con aproximadamente 30 pacientes" - http://scielo.isciii.es/scielo.php?script=sci_arttext&pid=S1... . The 59 occurs in, for example, "La reducción también fue estadísticamente significativa (59%) al tener en cuenta el coste del tiempo de enfermería".
#11 says "Pilot studies should include at least 12 patients [30,31], and cluster-randomized studies need more patients to correct for the clustering [32]. To have sufficient power for this pilot study, we intended to include 15 patients per nurse." where [31] is Viechtbauer. The value 59 is because "data of 149 (98%) patients were available of whom 66 (44%) were in the intervention group; ‡ = data of 135 (89%) were available of whom 59 (44%) were in the intervention group". - https://www.sciencedirect.com/science/article/pii/S026156141...
#12 says "In line with recommended sample sizes of pilot feasibility trials, 83–85 40 patients is deemed sufficient to explore feasibility, acceptability and potentially efficacy of the intervention, assuming retention rates of 80%, the true population consent rate will be with a margin of error of 13% (95% binomial exact CI)". The only "59" comes from citations. - https://www.sciencedirect.com/science/article/pii/S026156141...
I'm not sure how we're to have a take-home lesson when the false negatives so overwhelm the claimed evidence.
I'm also hung up on: "Capitalism and evolution work so well because of their merciless feedback mechanisms." because of obvious questions like "does capitalism work well for slaves?"
I prefer something closer to "An expert is someone who knows some of the worst mistakes that can be made in his subject, and how to avoid them". 'Cause even experts make mistakes in their own field.
This piece seems to think an expert cannot make a mistake in a given field of expertise, which is "utter nonsense, of course."
I don't like this example: "If you look at the papers citing Viechtbauer et al., you will find dozens of them simply using N=59, regardless of the problem they're studying"
Paging through the linked-to Google Scholar search finds 90 results. Of the first 12 with readable text, only two seem correctly described that way (and max 3), suggesting under two dozen examples. A spot check of the remainder suggests that pattern holds. I'll show my work:
#1 says "A sample size of 30 patients per group was chosen as recommended for pilot studies to achieve an appropriate level of statistical power [30]." They selected 60, and "1 patient was lost to follow-up" giving 59. So this wasn't N=59 but N=60. - https://medinform.jmir.org/2019/4/e14044/?utm_source=TrendMD... . 30 was chosen because "It corresponds to the detection of a potential difference of 21% between groups for a power of 80% and an alpha significance level of 5%." which is not the default example from Viechtbauer.
#2 does not use N=59. The number 59 occurs because: "The study included 16 patients with diabetes (59 ± 8.8 years; 8 men and 8 women)" - https://www.sciencedirect.com/science/article/pii/S095825921...
#3 uses N=59, though I don't know if it was used incorrectly because I can only read the abstract and citation list - https://journals.sagepub.com/doi/pdf/10.1177/0306624X1985848...
#4 specifically does NOT use Viechtbaueret, writing: "Viechtbaueret al., (2020) recommends sample size between 10-40 as adequate for feasibility study. Linacre, (2015) suggests a sample of thirty (30) respondents as appropriate for general purposes. This study however uses 60 respondents becauseadequate sample and item produce a good item, person separation and reliability in Rasch measurement (Fisher, Elbaum, & Coulter, 2010)." - https://journals.sagepub.com/doi/pdf/10.1177/0306624X1985848...
#5 uses N=43 ("With an alpha value of 0.05 and with an 80% power we need a sample size of at least 43 patients to detect a difference in validity and assess reliability") - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6886622/ . This is not the default example from Viechtbauer.
#6 uses N=59, using calculations directly from Viechtbaueret. It appears to be used correctly.
#7 uses N=53, commenting "There is no gold standard for sample size calculation in pilot studies. Sample sizes as small as 10 (Hertzog, 2008) and as large as 59 (Viechtbauer et al., 2015) have been recommended." - https://journals.lww.com/nursingresearchonline/FullText/2018...
#8 uses N=30 for each region, writing "Tamaño de la muestra: en cada región, se llevó a cabo un estudio piloto con aproximadamente 30 pacientes" - http://scielo.isciii.es/scielo.php?script=sci_arttext&pid=S1... . The 59 occurs in, for example, "La reducción también fue estadísticamente significativa (59%) al tener en cuenta el coste del tiempo de enfermería".
#9 points to the N=59 recommendation (correctly, for their experiment) but actually uses N=71 - https://www.mdpi.com/1422-0067/20/8/2024/pdf
#10 appears to have the discussed N=59 problem as it doesn't mention the 0.05 effect - https://mhealth.jmir.org/2016/3/e109/ .
#11 says "Pilot studies should include at least 12 patients [30,31], and cluster-randomized studies need more patients to correct for the clustering [32]. To have sufficient power for this pilot study, we intended to include 15 patients per nurse." where [31] is Viechtbauer. The value 59 is because "data of 149 (98%) patients were available of whom 66 (44%) were in the intervention group; ‡ = data of 135 (89%) were available of whom 59 (44%) were in the intervention group". - https://www.sciencedirect.com/science/article/pii/S026156141...
#12 says "In line with recommended sample sizes of pilot feasibility trials, 83–85 40 patients is deemed sufficient to explore feasibility, acceptability and potentially efficacy of the intervention, assuming retention rates of 80%, the true population consent rate will be with a margin of error of 13% (95% binomial exact CI)". The only "59" comes from citations. - https://www.sciencedirect.com/science/article/pii/S026156141...
I'm not sure how we're to have a take-home lesson when the false negatives so overwhelm the claimed evidence.
I'm also hung up on: "Capitalism and evolution work so well because of their merciless feedback mechanisms." because of obvious questions like "does capitalism work well for slaves?"