People make conclusions off of tinier sample sizes. There are plenty of scientific studies with just 50 or 100 people, but you need to keep the small numbers in mind.

As long as sampling was done correctly, then small sample size won't matter much !!

Most studies are worthless due to small sample sizes and confounding variables, and most “conclusions” are based on cherry-picked outliers. The fact that it’s common to do this doesn’t make it responsible or correct.

There's 15 documented adverse events, but we don't really know how many adverse events there were in total. If we simply study those 15 adverse events, then your population study is only those 15 events (with a hell-of-a-lot of selection bias).

----------

But I'm mostly thinking about how one person I know thinks: "Did you hear about these 15 cases and even one of them died! Clearly J&J is unsafe to take!!!!"

For that kind of conclusion, its like they've focused their mind entirely on 15 cases, as if their "population of study" was 15. Its obviously bad logic, but maybe I wasn't very clear in my original post that I was referencing some bad logic there.

---------

But there's plenty of 50-person or 200-person studies that ARE properly done, and the science is solid. Especially when we're looking at say Phase 1 / Phase 2 style trials of safety and/or efficacy (the earlier stage trials before a Phase 3 trial is conducted).

Its not a big enough sample to be "certain" of the results, but its a big enough sample to "provide evidence", possibly in conjuction with other studies.

----------

In many cases: having 5 or 6 studies with 50-random people each and performing a meta-study across them (especially if all 5 of those studies were of high quality sampling) would lead to a more robust conclusion than just one 1000-person study done slightly incorrectly (ex: the 1000-person "study" is a voluntary telephone survey or some other low-quality study)

This study referenced up above had the ~285 or whatever people who answered the survey, not that much info. ~285/350 is not that much data.

Anecdotally and based on my own experience I'd agree that the burnout is high, probably over 50%.

That is the value of a study of this size. The ability to quickly determine if there is any "there" there. The headline is sensationalistic but such is the nature of publish or perish science today.

Not even close to accurate.

If the election count is say, 48% for A and 50% for B and 2% other, that's that. You don't have error bars, you don't have student-t tests, you don't have random varaibles, you don't have "confidence intervals". You assume you have the ground truth. You have a precise number assumed to be without error.

Its actually a fundamentally different kind of math than what is typically taught in statistic classes (standard error vs standard deviation. R-values, random variables, etc. etc.)

When it comes to "total population" style information gathering, the questions of reliability are about whether-or-not you in fact, grabbed the total population... and nothing really "mathematical" (such as what are my 95% confidence interval?)

---------

When people say 600,000 died of COVID19 in the USA, there's no error bar, standard deviation, or random variable on that number. It is what it is: the ground truth according to a particular source.

And when you have a 2nd source of information with slightly different numbers (ex: State databases vs US level databases), you have a non-mathematical problem for how to resolve the apparent contradiction.

There's no "metastudy" over other surveys. There's no "resample again", there's no reproducibility. These statistical concepts and tests simply do not apply. If you do in fact decide to conduct a 2nd census, you will find that the numbers will differ in practice (ex: recounting the same set of ballots a 2nd time). But there's no underlying mathematical model for why this happens, as is the case in typical population statistics (student t-test assumes a standard error over a standard deviation of a hypothetical sample, etc. etc.)

In the case of J&J blood clots, there's billions of additional people who could take the J&J vaccine. 15/21 million people who took he J&J vaccine got blood clots. But if you want to know what percentage of "people who take the J&J vaccine get blood clots" that's not the same thing. The population includes an infinite number of people.

The converse also needs to be held in mind as well. Very large studies are more likely to display statistically significant findings. Your p-value is really just a test of your sample size, so as the study grows your p-value tends to shrink. Get a large enough sample and you'll almost always find statistical significance. Just referencing a p-value isn't enough, it needs to be read in the context of the study size (whether large or small).

Edit: For those downvoting, here is a much better and eloquent explanation:

https://twitter.com/daniela_witten/status/131218095960997478...

> If you’re testing whether mean=0 and actually the truth is that mean=0.000000001, and if you have a large enough sample size, then YOU WILL GET A TINY P-VALUE.

Think about this for a second. Does it make sense that the results of my statistical test would change if I measured in terragrams vs. micrograms? No, it doesn't, because it's not the numerical difference between null hypothesis and sample mean, it's the difference relative to standard deviation.

If the effect size is 0.000000001 and the standard deviation is 0.0000000000000001 then this is an enormous, important effect. If the standard deviation is 0.1, then this is a really small and unimportant effect. It can still be observed correctly with a sufficient sample size though because you've stated it's a real effect to begin with.

You will absolutely not get a tiny p value just because your sample size is large. That is wrong. You will be able to detect very small effects, but if there is no difference, you are more likely to draw the correct conclusion with 1,000,000 samples than with 100 samples.

The point the author here is trying to convey but is not getting clearly, is that a large sample size can detect an effect that is practically useless and call it significant. p-values tell you if it seems probable, given your data, if there is a difference between the two datasets. They make no claims about whether the difference is material.

Example python code to do prove p values are not likelier to be smaller just due to large sample sizes, given no actual difference:

   import random
   from scipy.stats import ttest_ind

   num_tests = 10000
   res = []
   for test in range(num_tests):
      n = 100
      a = [random.random() for _ in range(n)]
      b = [random.random() for _ in range(n)]

      res = res + [ttest_ind(a,b)[1] <= 0.05]

   print(f"10,000 tests with n=200 {sum(res)/ num_tests}")

   num_tests = 1000
   res = []
   for test in range(num_tests):
      n = 100000
      a = [random.random() for _ in range(n)]
      b = [random.random() for _ in range(n)]

      res = res + [ttest_ind(a,b)[1] <= 0.05]

   print(f"10,000 tests with n=20,000 {sum(res)/ num_tests}")

>You will be able to detect very small effects, but if there is no difference, you are more likely to draw the correct conclusion with 1,000,000 samples than with 100 samples.

I believe that this is to the tweet author's point where they stated that in the real-world a null hypothesis never exactly holds true. So you can detect even the tiniest of variance with a large enough sample.

>given no actual difference

I think the error here is that there is a priori knowledge of no difference, where the author is stating that in real-life scenarios there will always be some differences even if being too minute to be of practical significance. We can fabricate "no difference" in simulations but in real experimental design there will almost always be variance, even if it's just an artifact of the measurement process rather than being causal from the independent variable(s). Whether or not the differences statistically significant depends on the experiment design, to include sample size.

So while I understand your valid point I think the author's claim was more about the practical application of statistics rather than the mathematical precision as it relates with simulated examples. But I could be misinterpreting.

Of far greater problem is False Discovery Rate related things. Where you test 20 different things at once, and by chance identify one of them as significant even though the true effect size is 0. This is another area where increasing your sample size can help avoid problems, but even still you need to acknowledge your tools are imperfect.

I'm assuming you mean within the confines of the experiment, correct? I agree. The tweet author was eluding to the fact that "IRL" the null hypothesis is almost never true at the population level. Meaning if you grab a large enough sample you will detect very, very small differences. (This was her Lucky Charms ~ blood type example in the tweet). I also agree with that. I don't think the two claims are mutually exclusive and the fact they can coexist is (I believe) precisely her point about sample size.

Lucky Charms does probably relate to blood type in some impossibly small way. It makes sense that a biological trait has some relation to dietary consumption. I don't think any sort of non-garbage tier journal with peer review would publish it, but good on p-values for helping us detect them though. Not bad on sample size for making it possible to discern this effect size with a high degree of confidence.

It's worth noting that we also have many other tools to help us. For example you can test, given an expected effect size and a sample size, what the probability is of getting a statistically significant result, or a non-significant result, or a significant result that erroneously goes in the wrong direction. Or what the range of likely true mean effect size is given a significant sample difference.

We want large samples. They enhance confidence in findings. The author's premise seems to be it's better not to know that small rocks exist if we're only looking for big rocks. But fails to mention that the tools to find small rocks also help us identify big rocks with more clarity.

The variance of a Bernoulli distribution is P(1-P), which is (.83)(.17), since 83% of the respondents answered that they were burnt out.

I could go on and on.

It's not important whether the sun came out in the last two weeks or not, unless we were trying to measure the weather affecting burn out.

It's true different people might have a different interpretation of burn out, but it should be pretty similar to other people. I'm going to assume all people who took the study understand the idea behind burn out.

The point of the poll isn't to answer why but to measure the percentage of burn out with some degree of accuracy. The margin of error in this poll was 4.5%. That's how all yes/no polling works. We don't know that 83% of people are burnt out, but we are reasonably confident that 78% to 88% of people are burnt out.

Worth noting that the study itself doesn't seem to mention their sampling methodology.

I didn't think I was burnt out, until I started interviewing (due to work "stress", which turned out to be burn-out in disguise). One recruiter informed me they have seen a spike - 70% of interviewees canceled scheduled interviews, post-screening. I almost cancelled mine due to being overwhelmed/burnout from preparations + work, and my thesis is there are a lot of burned out developers out there. Unfortunately, I didn't hear what the cancellation rate baseline was before work-from-home.

If there are tech recruiters/HR folk/hiring managers reading this, I'd love to know the numbers you've been seeing for interview cancelations.

Often, the statistical methods are inappropriate at best, or downright deceiving at worst; there is nominal, if any, understanding about what "controlling for a variable" means, among many other such things. Not to mention that often the methods themselves are highly questionable in their reproducibility, statistics aside.

(Another classic is to point out flaws in generalizations for mice studies and such, but that is, imo, independent of the study's methods. It's a fair critique, but I don't think it's what you're interested in measuring w.r.t. "bad" studies.)

What makes you think data collected over 1 day worse than the same data collected in multiple days?

"I really don't understand how they can do a half-assed data survey on 258 people only, and then have the gall to claim "83% of DEVELOPERS""

It's called extrapolation.

No study ever interviews everyone. They all have to extrapolate from a subset of the entire population.

Obviously the more participants there are the more confidence you can have in the conclusion, but whether 200, 2000, or 20000 participants would make for a "good" study is completely arbitrary.

This only really works if you have a representative subset of he population you're extrapolating to. Is the distribution of experience roughly equivalent to the general population of "developers"? Age distribution? Company size? # of children? Home life? These kinds of demographics can have a big impact, and I see no discussion of how they were accounted for.

Heck, even the general work ethic culture difference between the UK vs the rest of the world could be a huge confounding factor that I know wasn't really accounted for.

Sure, a meta-analysis of several studies reveals more than one single study, but that doesn't make a single study bad or worthless - or as said in the parent - "a bullshit study".

This was a pilot study, a way to open an avenue of discussion, and to be sure I think the study warrants more investigation. They even say so: "In future we would like to conduct larger studies over multiple countries."

Of course. Never said otherwise.

while you may not intend that, most people reading your comment would believe you think the study is fine

No. I was replying to the specific objections the parent poster had.

I never said or implied anything about whether I thought it was a good study or not.

In particular, the way in which you mentioned extrapolation in the context of the study (and the conversation in general) carried the implication that the extrapolation was valid.

Wow. Several people on the internet think something.

Is that a large enough sample size?

It is, however, an extrapolation to make claims about the entire dev world when looking just at UK devs.

- Being common doesn't mean it's good or acceptable. In this case, it doesn't account for the effects of day of the week, or what sort of issues the person was having on that day specifically.

- n=258 is enough if it was sampled properly, and they provide no details to show that it was, so it wasn't. They provided nothing more than the engineers they surveyed were in the UK. Nothing on experience, age group distribution, family-size, years of work, company size, work change.

- As it stands, this provides 0 valuable information, and I can only assume it's intended for buzz.

I'm not clear on what you are suggesting here. Are you saying they should contact people multiple times to make sure they weren't having a bad day the first time around?

2-3 days is common because any longer than that and the survey results will no longer be reflective of a specific point in time.

Also, if you check their crosstabs they have breakdowns for gender and age.

If you have a really good, thorough study that is so thorough that people have nothing to comment on it, it will not rise to the top of pretty much any platform, because comments are valued higher than upvotes on most platforms. So you want to engineer some clickbait in there so that your article gets madly commented upon; being incomplete is perhaps the easiest way to do this. (I don't know about HN, but it certainly appears Facebook and Twitter work that way.)

nope not true, only algorithm driven website. just because this post or comment made it to the top of HN doesn't mean people think it's valuable now, it just means like you said: it has buzz around it, and it should, because it's bullshit and should be exposed for the bullshit it is. Is that enough to get people to be more skeptical about what they read? I don't know, probably not though.

Lots of social science seems to be confirmation of stuff that people have been suspecting. Not to take away from such things: It's important to verify with data.

However, the really interesting question: Where does the burnout come from? Is it possible to specifically characterize the source? By starting to break it down like that, we start toward finding a solution.

#depressing

Edit: Reality however: the opposite is true. Only in some scenarios (like reading under open sun) does one type of screen outdo the other. In most usual cases, that is indoor reading, the emissive display is better for the eyes.

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5929099/

There’s true burnout and then there’s feelings of stress. They aren’t the same thing at all. I know because I’ve experienced both. Burnout cuts you down to your soul. I haven’t programmed professionally in 2 years and the thought of doing so makes me nauseous. That’s burnout.

Thinking about it more, they can all be symptoms of the same thing, to varying degrees. Apologies to anyone I may have upset by minimizing their experience.

Psychological experiments and surveys are often done in only one or two days. Why would a wider window of time help when taking a snapshot of a current situation?

Hm, is 258 a small sample size? Let's put in some effort instead of just screaming "bullshit" a bunch. There are approximately 408,000 people in the entire population [0], and plugging everything into a calculator, we get two results: First, they should have tried to poll around 400 people (and perhaps they did!), but more importantly, the 95% confidence interval is only around ±6%.

Let me repeat that more clearly: By typical polling standards (95% confidence interval, random sample from filtered population), we expect that 77-89% of UK software developers are actually burnt out. That is still three out of every four people in the industry!

Not sure what to say. Take a stats class?

[0] https://www.statista.com/statistics/318818/numbers-of-progra...

"To reduce self-selection bias, invitations to complete surveys were sent out to members of Survation’s panel and the surveys were then conducted by online interview. This approach is far more rigorous than say, running a poll on a website in which anyone can choose to take part."

Are "members of Survation's panel" a random subset of developers? How many invitations were sent out to get 238 responses (i.e. what was the response rate)? Are developers with burnout more likely to respond to a survey about burnout? Was the sample population demographically similar to the overall population of developers?

That doesn't mean they are burned out clinically speaking, or even if they have symptoms of burnout. It does point to a large, systematic problem in the industry (and society in general).

I didn't claim to know everything there is to know about statistics and research in general, but I think I know enough to tell that this is not a study, but a marketing campaign for whatever this usehaystack thing is.

> First, they should have tried to poll around 400 people (and perhaps they did!),

If you had read the article, you would have known that that's not what they did.

> By typical polling standards (95% confidence interval, random sample from filtered population), we expect that 77-89% of UK software developers are actually burnt out. That is still three out of every four people in the industry!

Do I understand correctly that you've worked on this? That certainly explains the hostility, but nevertheless.

> we expect that 77-89% of UK software developers are actually burnt out.

That's not what the link/study says.

> That is still three out of every four people in the industry!

This is absolutely wrong, without a shred of a doubt.

I recommend that you read: How to lie with Statistics by Darrell Huff.