News for hackers. Stuff that matters.
Re:When Do Extraordinary Claims Give Extraordinary Evidence?... by nickpsecurity on Sunday December 06, 2015 @11:10AM (#10685565)
I've always found the "extraordinary claims require extraordinary evidence" meme ridiculous. It defies reason and the scientific method. Extraordinary claims simply need evidence they're true. That's it. Then, analysis and experiments should be conceived to determine why they contradict the "ordinary" model plus look for sources of error. If error or deceit are found, then the extraordinary claim is rejected. If not, the "ordinary" model is amended or rejected with data from new.

That simple. The need for extraordinary evidence seems to just promote orthodoxy. Better to focus on data and methods.

Re:When Do Extraordinary Claims Give Extraordinary Evidence?... by paulddraper on Sunday December 06, 2015 @12:17PM (#10685781)
In context, that expression refers to claims that contradict existing evidence.

For example, if someone finds a way to create mass/energy, that would be extraordinary. All existing evidence suggests mass/energy is conserved. Even if no one has tried XYZ before, the evidence of XYZ creating energy has to be weighed against the evidence of thousands of (other) processes not doing that. It's possible someone made a mistake, or the results are being interpreted incorrectly.

While there is an ideal, the reality is that "evidence" is of varying types and qualities, and that perfect evidence is not always feasible.

In the US, criminal convictions ("extraordinary" claims) require proof beyond reasonable doubt. But civil convictions ("ordinary" claims) require only a preponderance of the evidence.

Re:When Do Extraordinary Claims Give Extraordinary Evidence?... by andrewflnr on Sunday December 06, 2015 @01:08PM (#10685951)
That may be the ideal usage, but I'm not sure I've ever seen it used that way. If the phrase is about contradicting existing evidence, we should just say that instead.
Re:When Do Extraordinary Claims Give Extraordinary Evidence?... by nickpsecurity on Sunday December 06, 2015 @01:14PM (#10685980)
My point exactly. "Extraordinary evidence" just seems made up. I only require more and/or better evidence to back up an extraordinary claim versus claims contradicting it.
Re:When Do Extraordinary Claims Give Extraordinary Evidence?... by GFK_of_xmaspast on Monday December 07, 2015 @11:58AM (#10690685)
> more and/or better evidence

So, in other words, evidence that is beyond the ordinary. If only there were a prefix that could be applied here.

Re:When Do Extraordinary Claims Give Extraordinary Evidence?... by nickpsecurity on Monday December 07, 2015 @12:41PM (#10691047)
No, ordinary evidence or models that fit the observed data better or make more accurate predictions. I leave out the extraordinary part except for when praising scientists for talented observations or methods.
Re:When Do Extraordinary Claims Give Extraordinary Evidence?... by paulddraper on Monday December 07, 2015 @09:44PM (#10694439)
Okay. Well, here extraordinary is supposed to mean better/more then ordinary.

But if it means talented or praiseworthy or something similar, then I whole-heatedly agree.

Re:When Do Extraordinary Claims Give Extraordinary Evidence?... by gwern on Monday December 07, 2015 @10:46AM (#10690113)
So this is why you are a card-carrying member of a parapsychology association, right? Because many of the psi experiments, like Bem's recent ones, are better run than most psychology studies (sometimes much better run, as in, preregistered, highly powered, fully automated, and with adversarial collaborators running the study), and if you really believe that priors have no role in science, then you can't reject the psi studies without rejecting the rest as well.
Re:When Do Extraordinary Claims Give Extraordinary Evidence?... by nickpsecurity on Monday December 07, 2015 @12:48PM (#10691105)
It's why I changed my mind on occasion about various theories and why I was ahead of status quo on others. The secret? Focus on evidence and methods instead of where herd is going and how smart sources appear. Most of my beliefs are pretty orthodox because evidence supports them.

Your claims are mere trolling. Also, a perfect example of baseless beliefs posing as reason and without supporting evidence. Basically what you were accusing me of.

Re:When Do Extraordinary Claims Give Extraordinary Evidence?... by gwern on Tuesday December 08, 2015 @04:45PM (#10699807)
> Your claims are mere trolling. Also, a perfect example of baseless beliefs posing as reason and without supporting evidence. Basically what you were accusing me of.

No, it's not. Maybe you should look into psi research if you truly believe that you can believe something based solely on research standards. The problem with psi research is not that the methodology is bad, it's that it's so absurd that experiments better than 99% of all published psychology research still aren't enough proof.

So the question is, if you do not believe in psi, why not? It passes the criteria you have already laid out.

Re:When Do Extraordinary Claims Give Extraordinary Evidence?... by nickpsecurity on Tuesday December 08, 2015 @07:42PM (#10701035)
Well, if you were being serious, then all I can say is no comment on that as I'd have to look at recent findings, methods, and verifications. Looked into it a long time ago with the PEAR work being the most interesting. Needed duplication by independents that especially unlikely to work as a group to defraud people. Additionally, I proposed to people I discussed it with that, if psi phenomenon exists, skeptics' own negativity toward the experiments could prevent successful verifications. So, we'd have to have people who could run the experiment other skeptics constructed and approved of while being neutral or encouraging the subjects to create positive, mental state.

Not sure what's happened in the field since or if anyone else proposed something like I did. Radin & PEAR was only thing I saw then that looked like real science with well-thought methodology. Given high percentage of prior frauds, that not being ruled out in a substantive way in what I read on the topic had me saying "really interesting but who knows" as a result. I'd like to see the experiments re-run by diverse parties w/ conservative views on subject and reliable track record. Also, while mitigating both potential fraud and impact of negativity on test subjects.

My initial response to you came from you picking an extremely controversial field of research rather than examples of widely-held or supported beliefs countered with newer, better evidence that wouldn't be dismissed immediately by readers as psi topic would. I figured, like how those do with "conspiracy theory" labels, you were trying to get my comment dismissed entirely.

Btw, people are apparently calling out for you to respond in this thread about the Bitcoin leak:

https://news.ycombinator.com/item?id=10699846

Re:When Do Extraordinary Claims Give Extraordinary Evidence?... by gwern on Tuesday December 08, 2015 @09:02PM (#10701399)
> My initial response to you came from you picking an extremely controversial field of research rather than examples of widely-held or supported beliefs countered with newer, better evidence that wouldn't be dismissed immediately by readers as psi topic would.

And why is Psi so immediately dismissed? It's not because people have knockdown arguments about how sealed radioactive number generators can be easily biased to produce long runs or anything like that. It's dismissed based on priors: psi is just too absurd, contradicts just too much successful physics. And I agree and don't believe in psi in the slightest. Extraordinary claims really do require extraordinary evidence.

Re:When Do Extraordinary Claims Give Extraordinary Evidence?... by nickpsecurity on Wednesday December 09, 2015 @11:01PM (#10708794)
The priors are important. They can result in a dismissal but I'd put priors in context. Was the source unreliable? Was there a specific pattern in the research that led to unreliability before and exists now? Dismissing something because others were screw-ups or unethical would've been a good way to miss out on all kinds of scientific advances that amounted to rather simple, but new, experiments or data.

I'm all about priors being used to prioritize what verifiers focus on or to justify strong controls in event malice or error rates are a concern. Psi is a good example of both. Nonetheless, it's not "extraordinary evidence" I'm looking for as that's almost always an ill-defined term used for dismissal by groups being challenged. I'm just looking for reliable, repeatable evidence that fits the data better.

Let me give you an example that may or may not fit but feels like it does. A common attack in computers was stack smashing where incoming data flows toward stack pointer that controls execution, overwrites it, and causes rogue execution. Discovered and rediscovered in several ways over decades with 90's forgetting what was learned. Let's pretend that didn't exist & focus on mainstream (90's onward) approach. Their theory was that it was inherent and to be countered with either cheap tactics or expensive checking. Cat and mouse game for defence and attack followed with all believing that was inevitable and different people pushing different sides. That's a widely held belief, model, supporting claims, tests, techniques, and so on over thousands of people for decades.

Now, here's how you show they're all fools without extraordinary evidence or fringe science. :) You dust off a copy of the MULTICS security evaluations from 1970's to illustrate how it reduced risk of stack overflows by making data flow away from stack pointer and into new memory where available. Much better design decision & solves root issue everyone ignores. Other systems, esp hardware, tagged memory where they recognized what a stack was and had checks that turned attacks into mere error conditions. The root cause is simple to explain, the fixes are ordinary checks, the tests that they work are ordinary... everything here is simple, ordinary, and totally defies a whole industry worth of systems programming theory and practice. The "priors"... common wisdom & sociological factors here... prevent acceptance or adoption of this with many other tactics remaining. Although a select few radicals in academia and industry did implement it with effectiveness prior data showed.

That's what I'm talking about. We just do science as usual but with a different perspective, test, model, or whatever. Occasionally, one of these contradicts a lot of accepted beliefs or tests because they inherently biased themselves with narrow view or incorrect assumptions the new experiment doesn't have. So, people might re-run the data, check it, attempt to reject it, confirm it, whatever. And I still haven't found the need to define the concept of extraordinary evidence in any of this: just basic science and evidence.