News for hackers. Stuff that matters.
Re:Statistical Mistakes and How to Avoid Them by lisper on Wednesday November 23, 2016 @07:18PM (#13026907)
It's actually P(data|null-hypothesis).
Re:Statistical Mistakes and How to Avoid Them by qwrusz on Wednesday November 23, 2016 @09:04PM (#13027445)
How about you guys are both right, sort of.

There are both Bayesian and Frequentist approaches in statistics!

They represent very different methods to statistics, but they are also quite similar. My apologies, I couldn't find one link that gave a good description of Bayesian vs. Frequentist. Here are a couple links to get started:

https://xkcd.com/1132/

http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bay...

If anyone comes across a link that describes Bayesian and Frequentist clearly, please share it if you don't mind.

Re:Statistical Mistakes and How to Avoid Them by stdbrouw on Thursday November 24, 2016 @03:30AM (#13029049)
There are some cases when the difference between bayesian and frequentist statistics is just a matter of interpretation, but not in this case: the mathematics for a bayesian statistical test are different from the mathematics of a frequentist statistical test and they will produce different results, because the former accounts for prior probabilities and the latter does not. The article talks about frequentist tests, as does almost everyone else who mentions statistical tests without further specification.
Re:Statistical Mistakes and How to Avoid Them by aristus on Thursday November 24, 2016 @12:56AM (#13028490)
Frequentists count their chickens after they've hatched. Bayesians bet the chickens that they will.
Re:Statistical Mistakes and How to Avoid Them by lisper on Wednesday November 23, 2016 @09:16PM (#13027485)
That cartoon is actually a pretty good illustration of why frequentists are wrong.
Re:Statistical Mistakes and How to Avoid Them by qwrusz on Thursday November 24, 2016 @12:26AM (#13028377)
Do you mean probably wrong?

Either way I disagree. Both approaches have pros and cons depending on the type of analysis.

At the very least, just the presence of competing approaches in the field has pushed statisticians to have more rigour and do way more double checking than they might have, out of fear the other side actively looking to poke holes. It's easy to lie with statistics and the only people who can call statisticians out on their BS is other statisticians...Need more of this.

Re:Statistical Mistakes and How to Avoid Them by lisper on Thursday November 24, 2016 @12:58AM (#13028502)
> Do you mean probably wrong?

Nope.

> Both approaches have pros and cons

What are the pros of the frequentist approach?

Re:Statistical Mistakes and How to Avoid Them by rspeer on Thursday November 24, 2016 @02:57AM (#13028944)
You can justify that you did everything "by the book" and you don't have to make up priors.

Now. You might read this as me saying that Bayesian statistics is all made up, and that's not what I'm saying. I'm saying that if you go Bayesian all the way, and your result is even remotely controversial, someone could easily challenge you by saying "this result would have been different with different priors, and why won't they just come out and say p < 0.01? What are they hiding?"

When you do frequentist statistics, the equivalent of priors were, instead, made up for you by long scientific tradition. They're not very good priors, as the relevant xkcd illustrates. But at least it's not you making them up.

In a more complex model, there's also the thing where you can't calculate the result exactly, you have to approximate it with Markov Chain Monte Carlo or something, leading to another way to doubt your results (did the MCMC converge correctly?)

So instead you do frequentist statistics. You use your favorite stats package and it spits out a nice comforting p-value that will satisfy the reviewers. You have tons of guidance about how to do things. It's not a great thing, but it is a definite advantage of using frequentist statistics.

Re:Statistical Mistakes and How to Avoid Them by lisper on Thursday November 24, 2016 @05:07AM (#13029451)
In other words: the advantage of frequentism is that it gives you better plausible deniability when you get the wrong answer. Did I get that right?
Re:Statistical Mistakes and How to Avoid Them by rspeer on Thursday November 24, 2016 @11:09AM (#13030984)
No, and I don't know where you're getting your certainty that the answer is wrong from.

How about this: find me a published scientific paper with only Bayesian results in it, no frequentist statistics at all. Argue for its superiority all you want, but I don't think anyone does it. It would be seen as a stunt.

Re:Statistical Mistakes and How to Avoid Them by lisper on Thursday November 24, 2016 @04:57PM (#13033033)
Don't know about a paper, but here's a book:

https://www.amazon.com/Proving-History-Bayess-Theorem-Histor...

Re:Statistical Mistakes and How to Avoid Them by jcahill on Friday November 25, 2016 @07:17AM (#13036184)
> find me a published scientific paper with only Bayesian results in it

Trivial¹, surely?

> [B]ut I don't think anyone does it. It would be seen as a stunt.

What did you mean by "scientific"?

You'd have to hunt for a narrow reading s.t. the above holds — extant counterexamples aren't limited to any one branch.

Check it out² for yourself.

Further: cogsci's bayesian adoption is rapidly accelerating.

The replication crisis is brutalizing huge swathes of psych:

• loss of confidence in NHST is becoming near-total for many

• journals are purging in turn — e.g. BASP's p-value ban³⁴

• others are overhauling stats-in-psych entirely

J Math Psych alone has two recent special issues⁵⁶ on this.

I mean:

> no frequentist stats at all

isn't even a strawman lately, let alone an absurdity.

In some fields, it's a battle-cry.

All that being said… re:

> Argue for its superiority all you want

I wouldn't even go that far. Don't give em that.

Probability interpretation fundies

• are all wrong,

• narrow minds and waste lifespans with the cultism, and

• should at least learn of the other interpretations.

Pitching probability as a 1v1 isn't merely wrong-prime⁷ — it's doublepluswrong″⁸.

It's pseudofundamentalism: fundies uphold foundations.

Flamewars predicated on ignorance of the same gotta go, no matter how fashionable they may be.

____________________________________________________________

[1] http://enwp.org/bayesian_game

[2] http://google.com/scholar?q=fully+bayesian

[3] http://doi.org/4z8 | BASP 37

[4] http://doi.org/34p | Nature re: BASP

[5] http://doi.org/btqx | J Math Psych 72

[6] http://doi.org/btqz | J Math Psych 74

[7] http://enwp.org/all_models_are_wrong

[8] http://enwp.org/not_even_wrong

Re:Statistical Mistakes and How to Avoid Them by throw_away_777 on Thursday November 24, 2016 @08:12AM (#13030128)
The main advantage of the frequentist approach is that you can do the calculations much more easily. Bayesian statistics is great, but often the calculations are much more difficult because of your distribution of priors. You can make up simplified priors to ease the calculations, but then you run into some of the same problems as frequentist statistics.

Here is a simple example: lets say you flip a coin 10 times and get 8 heads, what is the probability that the coin is not fair? In frequentist statistics you only have to calculate the likelihood of the results for a null hypothesis, and then you use a p-value. While this approach is flawed, at least you can quickly do the calculation and get an approximate answer. In Bayesian statistics you have to specify the prior distribution, and calculate the likelihood of your results under every possible hypothesis. Correctly specifying this prior distribution and calculating the results is quite challenging - especially if you want to use a realistic prior (not just uniform). This is a pretty simple example, you can imagine how much more challenging this becomes in real-world problems. On the other hand, it is true that the frequentist approach doesn't really answer the question asked, so it is misleading (especially if you choose a p-value that isn't specific to the problem). If you choose p-values based on prior knowledge, than the differences between frequentist and bayesian are less extreme.

Re:Statistical Mistakes and How to Avoid Them by lisper on Thursday November 24, 2016 @09:42AM (#13030520)
A man was walking down a city street when he saw another man wandering around a lamp post looking at the sidewalk. "What are you doing?" the first man asked. "Looking for my keys," said the second man. "Oh, did you lose them around here?" asked the first man. "No," the second man replied, "but the light is better here."
Re:Statistical Mistakes and How to Avoid Them by throw_away_777 on Thursday November 24, 2016 @03:19PM (#13032515)
Often times getting close to an answer that is approximately correct is better than trying to find the perfect solution. Most real world problems, especially in analyzing data, don't have perfect answers. For example approximations are made all the time in physics, because without these approximations the calculations can't be done. Knowing when to make approximations and what approximations to make is an essential skill for analyzing data.
Re:Statistical Mistakes and How to Avoid Them by ska on Thursday November 24, 2016 @09:26AM (#13030443)
> Bayesian statistics is great, but often the calculations are much more difficult because of your distribution of priors.

This is often a feature, forcing you to actually look at the complexity head on before you sweep it under a rug.

Re:Statistical Mistakes and How to Avoid Them by danieltillett on Thursday November 24, 2016 @01:49AM (#13028685)
Simple maths that can be done without a computer. Useful in the past - not so useful today.
Re:Statistical Mistakes and How to Avoid Them by jcahill on Thursday November 24, 2016 @04:59AM (#13029417)
All models are wrong. That much is entailed by 'models'.

There is no Zuul.

Re:Statistical Mistakes and How to Avoid Them by wfunction on Wednesday November 23, 2016 @07:30PM (#13026966)
The model is the null hypothesis here... I don't think the parent comment was wrong w.r.t. this.
Re:Statistical Mistakes and How to Avoid Them by lisper on Wednesday November 23, 2016 @08:02PM (#13027091)
No, there's a difference. The model is something like, "This drug attaches itself selectively to cancer cells and kills them." The null hypothesis is, "This drug has no effect." So you conduct a double-blind study, measure the effect of the drug on cancer cells, collect some data and compute that P(data|null-hypothesis) is 1%. It it not the case that there is a 99% chance that your model is correct and that the drug does in fact attach itself to cancer cells and kill them. Because there are other possible models, e.g.: the drug redirects your Chi and channels it into your chakras which kills the cancer cells. Statistics alone cannot tell you which of those two models is correct.
Re:Statistical Mistakes and How to Avoid Them by nonbel on Wednesday November 23, 2016 @08:21PM (#13027174)
>"Statistics alone cannot tell you which of those two models is correct."

Statistics can tell you whether a model is consistent with the data. But you need to deduce the null hypothesis from your model rather than use the default "no difference" (of course, sometimes no difference is deduced from a real model, but not often, in that case: great!).

In fact, that is the proper use of statistics. I would guess >99.99% of current usage is incorrect (ie pseudoscience) and amounts to a waste of time at best. The usual usage turns scientific reasoning on its head, and has lead to a (literally for most people) unbelievable amount of trouble.

This was pointed out most aptly by Paul Meehl long, long ago: http://www.fisme.science.uu.nl/staff/christianb/downloads/me...

Re:Statistical Mistakes and How to Avoid Them by lisper on Wednesday November 23, 2016 @09:03PM (#13027438)
> Statistics can tell you whether a model is consistent with the data.

Yes, that's true, but it badly misses the point. The power of statistics is to tell you when a model (the null hypothesis) is (most likely) inconsistent with the data so that you can confidently rule it out. Any finite data set is consistent with an infinite number of models, so knowing that a model and the data are consistent tells you absolutely nothing about whether or not that model has any relationship with reality (which, at the risk of stating the obvious, is what science actually cares about). This is the reason that rejecting the null hypothesis is considered a positive result.

Re:Statistical Mistakes and How to Avoid Them by ACow_Adonis on Thursday November 24, 2016 @01:21AM (#13028586)
Except such a data set is also inconsistent with an infinite number of models, so ruling one out via rejecting a null also provides practically no information value and moves us no closer to understanding. /devils advocate

In practical terms, we're not interested in true models, but useful ones, so the description of a model's consistency with observed data is often the more useful metric in practice than rejecting nulls :/ especially in applications where you can't set up repeated experiments.

OK, I realise its more nuanced than that too, but given how many papers and practitioners seem incapable of understanding that evidence against the null it's not explicit evidence for an arbitrary alternative, practically and consequentially I don't think that's how we should be working...

Re:Statistical Mistakes and How to Avoid Them by lisper on Thursday November 24, 2016 @05:14AM (#13029482)
> Except such a data set is also inconsistent with an infinite number of models, so ruling one out via rejecting a null also provides practically no information value and moves us no closer to understanding.

No, that's not true, because experiments are not done in a (figurative) vacuum. They are done in the context of an explanatory theory that has already gone through a vigorous filter and shown to be consistent with the all prior experimental data and has better explanatory power than all of its competitors. It is only when more than one theory survives this filter that an experiment is done, and the experiment is designed specifically to distinguish between the surviving theories.

So while it is true that an experiment allows you to eliminate an infinite number of theories, it's irrelevant, because by the time the experiment is done nearly all of those theories have already been eliminated anyway.

Re:Statistical Mistakes and How to Avoid Them by nonbel on Thursday November 24, 2016 @10:36AM (#13030822)
>"The power of statistics is to tell you when a model (the null hypothesis) is (most likely) inconsistent with the data so that you can confidently rule it out"

If you know whether your model is consistent with the data, you know whether it is inconsistent... I think you are talking about some other issue than I am.

The point about deducing the null hypothesis from your explanatory model is that the null hypothesis is precise. In that case you will get a strong test of the model, and it will get stronger as more data gets collected. Using a default null and and vague alternative is the exact opposite. (check the Meehl 1967 paper I linked earlier in this thread).

Re:Statistical Mistakes and How to Avoid Them by lisper on Thursday November 24, 2016 @06:11PM (#13033380)
> If you know whether your model is consistent with the data, you know whether it is inconsistent...

No, that's not true. And in fact I got it wrong earlier when I agreed with you that statistics can tell you when data is consistent with a model. They can't. At best they can tell you whether the data are not inconsistent. That sounds like the same thing, but it isn't. It's like the distinction between "not guilty" and "actually innocent." At best (or at worst depending on how you look at it) a statistical test can tell you, "This theory cannot be confidently ruled out on the basis of that data under the following assumptions..."

> I think you are talking about some other issue than I am.

That is quite possible.

[UPDATE:] BTW, I read the Meehl paper, and I completely agree with what he says. So you and I may be in "violent agreement" here.

Re:Statistical Mistakes and How to Avoid Them by fluxion on Wednesday November 23, 2016 @08:17PM (#13027153)
I think this is just an argument over semantics, the "null hypothesis" is a perfectly valid model using the definition of the statistics community [0] (i.e. a collection of probability distributions over some sample space).

[0] https://en.wikipedia.org/wiki/Statistical_model

Re:Statistical Mistakes and How to Avoid Them by lisper on Wednesday November 23, 2016 @09:12PM (#13027472)
The word "model" is being used here in two mutually incompatible ways. Yes, the null hypothesis is a model, but it is not an explanatory model. A scientific hypothesis has to meet two tests to be considered a valid theory. It has to be consistent with the data, and it has to have explanatory power. The theory that cancer drugs work by aligning a patient's chi with their chakras is rejected not because it is inconsistent with the data (it's not) but because it lacks the explanatory power of alternative theories based on molecular biology.

The null hypothesis never has explanatory power. The null hypothesis is always a statement of the form, "The explanatory hypothesis under test is wrong for some unknown reason." This is why rejecting the null hypothesis, i.e. showing that the data are (with high probability) inconsistent with the null hypothesis, is considered a positive result.

Re:Statistical Mistakes and How to Avoid Them by stdbrouw on Thursday November 24, 2016 @03:23AM (#13029024)
Yup, and the null hypothesis is a reduced model. Different words, same thing.
Re:Statistical Mistakes and How to Avoid Them by lisper on Thursday November 24, 2016 @05:17AM (#13029496)
I have no idea what you mean by "reduced" but they are not the same thing. The difference in their explanatory power. A (scientific) hypothesis explains things. A null hypothesis simply says, "This hypothesis is wrong" but doesn't say why. It's a crucial distinction.