Thank you for this! I get frustrated with the "N of only 21? Might as well flip a coin!" responses. Like you say, the whole purpose of statistical testing is to give an accurate, numeric value that says how likely the results are due to chance.

One thing I'd note, though, is that the paper's title is "High sensitivity saliva-based biosensor in detection of breast cancer biomarkers: HER2 and CA15-3". My understanding is that sensitivity was never really the problem with breast cancer detection - it's specificity that is the real challenge with all types of broadly-deployed medical screening tools.

Sensitivity matters in this case because the medium is saliva which contains a fraction of the antigens contained in serum. Specificity is challenging when the biomarkers are non-specific and you only have 1 testing modality but that seems to be not so much the case here.

Well, depends. It’s bayes law fucking us over with the vast majority of people not having breast cancer but really, really needing to know if they do.

This is why it’s better to have tiers of confidence for actual policy decisions and not just confusion matrix results frankly.

Like if you run this test and you’re in a cohort that is 99.999% likely to be cancer, that’s useful info. It’s not a problem, per se, if the test instead comes back with an answer of mixed certainty. We just need to be comfortable with getting neither a positive nor a negative prediction; nor expecting it to cover all cases.

The policy driving thresholds on tests need to consider the weights of each possible outcome to determine what to do.

No matter what the value of N is, someone always comes out of the woodwork to complain about it. It's a pretty common criticism of any study that relies on statistics, and one anyone can make in a few sentences.

People on HN (maybe elsewhere too) devote a lot of attention to sample sizes. I don't know the upthread commenter at all, but in general I suspect it's because that is an easy thing to understand about research and statistics, and it's a valid critique they've seen professionals use.

A normal human fallacy is to focus on the thing you understand, that is easy to understand (e.g., easy to quantify), and overlook the difficult issues that are far more important. There is much more, of much more importance going on in most of these studies, and in their statistics and validity, than the sample size.

Elsewhere too

I first took stats in high school and we read a bunch of valid small sample size studies so I don’t know where people are failing to get educated.

There were so many factors going into whether a study was good.

Exactly.

Famous example: Lady tasting Tea [1]. N=1 or N=8 depending on how you look at it. Still significant.

This is the one that proves that it's possible to tell whether you added the milk first or second.

[1] https://en.wikipedia.org/wiki/Lady_tasting_tea

This is why science has gotten harder these days. When there were only half a billion people, sampling 500 got you 1 in a million coverage. Now there are 8 billion people, so it's worth 16 times less. As more people are made, science will suffer because the percentage is so much less. In fact, this is why I always trust science in small towns more. In Colma, for instance, you sample 500 and you have covered 33% of the people. Sometimes, it's even better to just sample people in one house. 100% coverage.

TBH, I can't tell if this is sarcastic or not, because there is so much in this comment that belies a fundamental misunderstanding of statistical testing and statistical power, which is the point I was trying to make.

> Now there are 8 billion people, so it's worth 16 times less.

That is not how statistics works, at all.

> In fact, this is why I always trust science in small towns more. In Colma, for instance, you sample 500 and you have covered 33% of the people.

These are the sentences that made me think this had to be satire (presumably you want "science" to apply to places outside of Colma...), but in all honesty these days it's really hard to tell.

My assumption is that 98% of people have no idea what the right sample size is for this particular experiment, including myself of course.

To all who criticize and ridicule someone who would like to have more samples, why do you think 21 is such a perfect number in this case? Wouldn't 15 be enough, if statistics and all applies? 10? 1? Would 30 be too much?

> To all who criticize and ridicule someone who would like to have more samples, why do you think 21 is such a perfect number in this case? Wouldn't 15 be enough, if statistics and all applies? 10? 1? Would 30 be too much?

This is the point that you are fundamentally misunderstanding. Nobody is making the argument that "21 is such a perfect number in this case". The rest of your sentences ("Wouldn't 15 be enough, if statistics and all applies? 10? 1? Would 30 be too much?") seem to point to a belief that people are pulling numbers based on a finger in the wind.

The whole point of (a lot of) statistical testing is that it allows you to come up with a specific number that determines how likely a particular result is due to chance. That is what the p < .05 "standard" is about - it's a determination that the results have less than a 5% likelihood to be due to random chance (though that 5% number used as "significant" is just basically pulled out of thin air, and p-hacking is another topic...) That is, I and the comment I replied to aren't making the argument that 21 "is such a perfect number". We're making the argument that even with a small sample size it's possible to determine the relative error bars, with precision, using statistical methods, not just pulling a number based on feels. Yes, larger sample sizes reduce the size of those error bars. But often not in ways that are "intuitive". You have to do the math.

None of what I wrote above is meant to imply that statistical tests can't be misused, or that they often require assumptions about the underlying population distribution that may not be correct.

It is a function of the effect size, not the experiment. When the effect size is large, you need fewer samples to detect it reliably. In this case it looks pretty large. Would you want more people before rolling it out? Of course. But it’s very unlikely to be vaporware provided the samples are random.

Thank you! Came here to say, but found, that you've already written it.

Effect size is going to be important when talking about clinical significance, not just statistic significance, too.

I also want to point out that we have no stats on sensitivity, specificity, or diagnostic odds ratio, which are all clinically relevant to physicians deciding when to test and how to interpret test results.

The good news is that it's a non-invasive, low cost test which plays a factor in clinical decision-making.

If you only flip a coin to the same side 21 times in a row, it could just be chance!

It's only 1 in a million, but at about 10 the same I'd start to figure the coin wasn't fair.

And this result probably helps the funding to get more patients. I’ve never worked in clinical trials but it seems like it would be so tiresome.

I ran many clinical trials during my career as an academic neurosurgical anesthesiologist. "... it seems like it would be so tiresome" is accurate; it's also exhausting in terms of the huge amount of time and effort required to get a study approved by the institutional review board; getting informed consent from prospective patients after an exhaustive explanation repeated over and over to each individual; actually doing the study; organizing the results; doing the statistical analysis required; writing the paper; waiting months to hear back from the journal's reviewers, often receiving a rejection letter; resubmitting the paper to another journal, sometimes several more; after having it accepted, revising the paper per the reviewers' comments before resubmission, sometimes going back and forth for months.

You can find my published papers here:

https://scholar.google.com/citations?user=5DdrMc8AAAAJ&hl=en

So how do you feel about it? :) Seriously, what are you feelings about it after all those experiences?

You forgot the end of the story: After all that, it's done and published, and then a random person with no expertise and who barely read the paper posts on HN: the sample size is too small - as if you were in your first week of statistics 101 - and therefore the whole thing must be invalid! :)

How do I feel about it? My feelings after all those experiences?

I dunno... that was me then, I guess: hard core academic with a drive/compulsion to publish good work.

I admit to being amused by comments here about statistics by people who wouldn't know a Bonferroni correction from a bonfire.

With so few controls, it’s still not a well designed comparison. Most scientists deal with low sample sizes, especially in first trials. However, what is often overlooked is the need for sufficient numbers of negative controls. If they only had 4 control patients, that’s completely inadequate to draw any real conclusions. With access to a biobank at U of Florida, they should have been able to test more samples — especially with a $5 test.

There are two other issues with this paper [1] that I quickly see. Their main figure lacks errors bars. It’s pretty clear to me that the groups would overlap quite a bit. More numbers would make this problem clearer (either to narrow error bars or clearly show an overlap). The lack of error bars across the paper make me think they didn’t do any technical replicates, which is also a problem.

I’m also not sure a one way test is correct here, but I’m also not entirely sure how they are measuring the data. In this one way analysis, all you can tell is if one group is higher than the other. When the data are so unbalanced, what you don’t see is what the predictive value of the test is — false positive/negative. That’s the real issue here. It looks like you’d have a really high false negative rate, as the cancer samples have a much wider range than the controls. This is the worst thing you can have in a test like this.

Finally — this paper was published in the “journal of vacuum sciences and technology B.” There is no way this paper got a valid peer review to make these claims. I don’t know anything about this journal, but I doubt it has much experience with cancer testing.

[1] https://pubs.aip.org/avs/jvb/article/42/2/023202/3262988/Hig...

A small control sample is not “few controls”. Misleading term. “Negative controls” is not a thing either. Just the control group is fine.

The error bars are not really needed. Your eyeballs are doing just fine. Yes, they overlap. Yes, it would likely have a high false negative rate.

A false negative rate is not a problem for a cheap test. This test is not meant to replace higher quality and more invasive tests. A high precision, low recall test still has significant value. You merely have to accept that a negative result tells you very little.

If we imagined for instance, that the false positive rate is very low despite the overwhelmingly larger population of people without cancer, then this would be of enormous value.

This is an unbalanced study. As such, it doesn't tell you anything about the ability to differentiate between the three populations. You can argue about the terms "negative control" (which is appropriate here, as there is a positive control test in the paper), but there are only 4 non-cancer samples tested. That is not enough to be able to adequately know range of measurable values in the population of patients w/o cancer.

But, that's not really the point. They aren't trying to diagnose cancer vs. healthy.

Error bars here are absolutely necessary. Two reasons: First, you want to know the approximate ranges for each group in Figures 3 and 5. Not showing them is misleading. Secondly -- you actually also want error bars for each patient sample. I'd expect for there to be at least three replicates for each saliva sample to show that the strips are able to consistently measure a known value from each sample.

I also mis-read part of the paper the first time. For the HER2 cases, there aren't 4 negative samples -- there are 20. There is only one positive sample. Part of the problem is really how they are presenting the data -- it is not all clear what they are testing. But, there is only one HER2+ sample in the mix.

One... N=1.

Samples include:

* Non-cancer: 4

* In situ cancer: 3

* Invasive cancer, HER2-: 13

* Invasive cancer, HER2+: 1

What you'd really like to show is that the HER2+ patients could be differentiated from HER2- patients. Which, does look really good, but with only one HER2+ sample, you really can't tell much. (And the presence of so much signal in the HER2- samples raises some very interesting biological/mechanistic questions).

Note: I'm not trying to say that the authors of the study are wrong or are trying to deliberately mislead people. There is so much here that could have been corrected to make this a much stronger paper. To me, this seems like a paper where the authors are likely engineers and not that well versed in biomedical statistics. The paper is published in a physics journal, so the journal itself is not a good place to make some of these arguments.

Is the idea of a non-invasive test worthwhile? Yes! Absolutely. But they didn't show that it was a good test of clinical utility. They showed that it could measure differences in protein concentrations from saliva. That's not nothing, but that's it. Now, if that is an appropriate way to differentiate patients is a completely different question and requires substantially more testing (and orders of magnitude more patients).

> This is an unbalanced study. As such, it doesn't tell you anything about the ability to differentiate between the three populations.

Complete Nonsense

> That is not enough to be able to adequately know range of measurable values in the population of patients w/o cancer.

True. But it is a promising initial signal that the distribution of non cancerous folks is probably very different from the invasive cancer folks. The effect size here is huge. “Range” is less interesting than “Distribution”

> Error bars here are absolutely necessary. Two reasons: First, you want to know the approximate ranges for each group in Figures 3 and 5. Not showing them is misleading.

You don’t really need error bars when you’re showing all of a small number of data points. But sure whatever

> Secondly -- you actually also want error bars for each patient sample. I'd expect for there to be at least three replicates for each saliva sample to show that the strips are able to consistently measure a known value from each sample.

Would be nice. Sounds like they did ten measurements per.

> What you'd really like to show is that the HER2+ patients could be differentiated from HER2- patients. Which, does look really good, but with only one HER2+ sample, you really can't tell much. (And the presence of so much signal in the HER2- samples raises some very interesting biological/mechanistic questions).

I’m not clear why you’re saying HER+ vs HER- is the important difference here.

Look, you obviously have your opinions here. I'm not sure just saying "Complete nonsense" at things is really all that helpful.

What I said ("it doesn't tell you anything about the ability to differentiate between the three populations") is quite correct. This study shows that there is a difference between the groups of samples tested with their HER2 test strip, with a one-way p-value of ~0.002.

I'm not convinced that the samples are representative of their populations. The number of non-cancer samples too low.

>* You don’t really need error bars when you’re showing all of a small number of data points.*

Error bars are visually helpful ways to show that the group values overlap. Which, in this case, they do (I did replot this data to confirm).

>* Would be nice. Sounds like they did ten measurements per.*

This is for one test. They sampled the test strip 10 times. I mean they should have tested each sample on at least 3 different test strips to get a mean value for the sample. This is a paper that is trying to say that their test strips are accurate, so it would make sense to test them multiple times.

>I’m not clear why you’re saying HER+ vs HER- is the important difference here.

I'm not sure what they are trying to claim in there paper... are they trying to say that they can diagnose breast cancer (which would requires many more biomarkers), or are they trying to say that they can differentiate between HER2+ and HER2- cancers (which would be more appropriate for a HER2 test).

The other biomarker has even more overlap, so not sure how helpful that would be.

Really, I think they are also missing an opportunity -- the bigger use for me would be in longitudinal testing. If they could show changes in signal over time for a particular patient that corresponded to treatment status -- that would be a great use for a cheap non-invasive test.

> Look, you obviously have your opinions here. I'm not sure just saying "Complete nonsense" at things is really all that helpful. > What I said ("it doesn't tell you anything about the ability to differentiate between the three populations") is quite correct. This study shows that there is a difference between the groups of samples tested with their HER2 test strip, with a one-way p-value of ~0.002.

You claimed this was the implication of an unbalanced study. I’m sorry but that is a complete non sequitur. It is nonsense. Most of everything else I disagree with but isn’t nonsense.

> I'm not sure what they are trying to claim in there paper... are they trying to say that they can diagnose breast cancer (which would requires many more biomarkers), or are they trying to say that they can differentiate between HER2+ and HER2- cancers (which would be more appropriate for a HER2 test).

They are simply showing different distributions of tests for different populations and observing that, foremost, the invasive cancer ones have a significantly different distribution; and that it’s shows promise as a test for the future.

It's been a while since I took statistics, but isn't the whole point of <thing> testing that it needs to have a high Bayes factor, and to be confident that said Bayes factor is high?

In this case, and my lack of understanding in both bio and stats is showing here, we're trying to develop a test for breast cancer. An ideal test will have high sensitivity and specificity, and a tight confidence interval for both of those numbers. This way we can be confident that a positive/negative test actually moves our priors meaningfully in either direction.

I guess my question/comment is more on the fact that I don't see how any of the results shown actually translate, as the headline suggests, into a cancer test of high power. The priors for any kind of cancer is pretty low from what I can find, so we need high power tests in both the positive and negative direction to meaningfully effect health outcomes. I can't find any CI numbers in the paper, which may just be me not reading it closely enough, but it doesn't help my confidence.

Suppose you could roll a die, and if the die came up six, it would tell you with high certainty you had cancer, and if it came up as anything else, it would tell you nothing.

If the test is very cheap, it’s probably a great test presuming you get to see the dice roll itself.

The cost and invasiveness of the test is important.

The problem here is that the underlying and unknown correlations in the sample (aka people) is that they are NOT independent.

The larger sample sizes are a ... sort-of "proxy" ... to overwhelm underlying latent correlations.

The whole thing is actually a subtle sort-of generalization of the "Prosecutor's Fallacy".

So skepticism with the small sample sizes is absolutely warranted, unless some strong evidence is shown indicating mechanism-based independence.

I feel like you’re taking a very roundabout way to describe the concept of sample bias. So long as it’s a random sample, this is accounted for in the statistics. Yes, we should still get more data all the same.