"I could improve your ultimate financial welfare by giving you a ticket with only twenty slots in it so that you had twenty punches - representing all the investments that you got to make in a lifetime. And once you'd punched through the card, you couldn't make any more investments at all. Under those rules, you'd really think carefully about what you did, and you'd be forced to load up on what you'd really thought about. So you'd do so much better."

https://www8.gsb.columbia.edu/articles/columbia-business/sup...

The way Buffett invests is by reading all day, everyday and once in a while making a very informed investment. Most people haven't even reached the level of staying away from 'sure things' and panic selling during bad news.

How many people would've been able to stay away from IT stocks during the dotcom boom, to even not invest in the company of one of your friends. (Bill Gates and Buffett have known each other since '91)

How often do you, as a 'master', break rules that you would tell a 'apprentice' are immutable?

Of course that's just one form of risk. The less attractive aspect of a 20 stock strategy comes from the fact that the majority of the market's returns come from very few stocks - The 80/20 rule applies pretty well here. With only 20 stocks you'll probably miss out on the few winners that contribute all the market's gains. It's very easy to end up with a 20 stock portfolio with low risk/variance and low returns.

Of course if you are Buffet, then your goal is to pick 20 stocks that all outperform.

BTW, the lack of new results is actually a good thing. It means we've got a lot of stuff figured out.

Of our current energy/resolution paradigms

We know that ~20% of the universe is Dark Matter and that ~75% is Dark Energy. All we really know about Dark Matter is that it falls down and doesn't really interact with itself or anything else, really.

Dark Energy? Yeah, whatever it is, it pushes galaxies apart, that's about all we got.

There are a LOT of pretty easy discoveries to be made yet, but we just don't have the engineering to make them yet. The LHC was a marvel of engineering, but it can't even come close to the energy required to produce (theorized) Dark Matter particles. If you tried to make a bigger one, you'd need an LHC with a radius larger than the Earth's. That's not going to happen anytime soon, not by a long shot. We need another method than particle colliders, probably a very large space-based telescope to stare at black-holes with jets.

Biology is also rife with 'easy' things to do, but ones that we just don't have the tech for yet. Strangely, with all of the data that the Googs and the Zuck have, we have not seen a similar paradigm shift in psychology, criminology, or economics. At least in public literature.

So, maybe it's not so much that we have stuff "figured out", as it is that we have mental models that, while they work well enough to make certain predictions, also constrain our thinking in terms of what phenomena are possible. The "EM drive" controversy is one example of this tension.

I'm not so sure of this. Have you heard of Edward Leedskalnin, or his "Perpetual Motion Holder"? The textbooks that I have discussing magnetism do not explain this effect, which you can see an example of here: https://www.youtube.com/watch?v=eWSAcMoxITw

If you can explain this effect with our "current theories", I would love to hear it!

There's another interesting effect where an electric charge can be pulled from that set-up. Lasersaber demonstrates that here: https://www.youtube.com/watch?v=S_ssUTRbRRs

I certainly can't explain it, and I don't think we have an adequate theory that does explain it (yet).

In other words, much of the problem we face in the current scientific age is due to the fact that our instruments are now electron microscopes and Hadron Colliders, instead of carefully made glass beads and simple mirrors.

It's similar to problem faced in software of managing increasing complexity. No silver bullets known of yet for all those problems, just chronic application of powerful minds to the problem. :)

For example within medicine when computer power and AI is advanced enough to help us much more connecting the dots between DNA, viruses, environment etc.

https://en.wikipedia.org/wiki/Superconducting_Super_Collider

for posterity: That collider can't get to the energy levels needed because it doesn't exist. It doesn't exist because it's construction was cancelled after the project was defunded. Where it to have finished construction, it would accelerate protons up to 20 TeV. The LHC reaches 6.5 TeV. All of that information is clearly stated in the link I posted and is completely uncontroversial.

Stop bullshitting.

Not necessarily. If anything, the more we find out, the more we realise how much we don't know. Take quantum physics, that stuff slowed down since the 1930s.

It's not a bad thing, the journey's always fun too. But let's not kid ourselves that we're close to figuring everything out.

And this shouldn't really be a surprise. Our cultures and societies have had thousands of years of evolution; our human instincts have had at least hundreds of thousands of years. In that time, both culture and instinct have picked up many true facts about the world that have helped us survive. A lot of science has been taking those facts and systematically investigating how far they hold.

So these both can be true: the more questions we ask, the more questions we have to answer, but also that many of the 'easy' hypotheses have been mined out, and any true advance in science will by necessity pull us further and further from the realm where our basic instincts and intuitions can usefully guide us.

Quantum mechanics being the prototypical example, of course.

Like a tree? The main branches are the most visible, and so the most tackled. The smaller branches are only realised when you're deep enough in it. That's where our age's advantage comes in, as we have many more qualified 'explorers' than before, but it also makes it all the more important that the foundational branches are solid.

But then again it may be closer to readjusting our lens. The big questions (of the natural kind, not innovation like AI) today still pertain what we take for granted, like quantum mechanics is making us question the nature of reality again. But much like the discovery of waves once threatened the credibility of light as particles, I'm optimistic that reconciliation can happen, and to ascend to another level ... but where can we stop? When can we say "This is it"?

Just musing..

We are, clearly, close to figuring out most of the easy stuff. For our current definition of "easy" based on the amount of work, computing power and strength of the mathematical tools that we consider normal.

There really are very few edge cases left for QM 2.0

I'm not sure about that. As a former working scientist, I would say that the primary driver of this is the promotion and funding structure of science. I do agree thta a lot of low-lying fruit has been picked.

Well, yeah, obviously. I'm saying that structure is there because it's a holdover from previous generations where novel results were easier to obtain.

http://www.thenewatlantis.com/publications/saving-science

Genuinely new results require developing new insights, new techniques or just making new guesses that happen to be right. All are rare enough. But once that is done, the information gets out and the experiments should be easily reproducible.

Some dispciplines get a lot of reproduction as a side effect of using last year's breakthrough research as "mere" technology for implementing next year's expriment. But I guess that sort of thing is less natural pyschology and sociology.

And it's possibly more costly than the original since the initial lab likely was able to re-use some of their existing resources, trained people, infrastructure, lab animals, equipment (whatever matters for the discipline) which you don't have; they likely chose this particular experiment because it fits well with the stuff that they have that's not available everywhere. If they developed a particular experimental setup / gear / network of volunteers doing interviews in the subpopulation you're studying / cell or animal line / etc, then they're using that for many publications over the years, but you need to spend the whole cost to reproduce any one of them.

I was once an experimental physicist and never psychologist. In physics there is a lot of trial and error before you can get the experiment to do anything at all. Anyone who reads your paper needs fewer trials.

So I was going to agree with you that in fields where experiments work differently, reproduction can cost as much as the original study. But on second thought, that's wrong: and indeed it is the whole point of the thread.

There is always trial-and-error. You simply don't know if your theory is correct. And there is always risk of a methodolical weakness that you only understand after seeing the data. The trouble is after you have already spent all your time and grant money on your study, you have every incentive to paper over these issues. Indeed even if you are scrupulously honest, you will have to shove study in the proverbial file drawer - because you don't have the resources to try another iteration.

So I think it's still true that a new experiment costs more than reproduction, and many of the problems with studies arise from trying to pretend that cost doesn't exist.

It's hard to find a Roman gold coin in farmer's field but once you've found your first one you can be more confident that the second will be easier to come by.

Easier, if you factor in how much easier P hacking is with modern software.

We don't know anywhere near "most" things. The supply of novel discovery is not the limitation.

Please don't use that quote. It's apocryphal; he almost certainly never said that.

It's not about how close we are to "knowing most things," but about how accessible new learning is. In a game of musical chairs, all the easy chairs are taken.

> The beauty and clearness of the dynamical theory, which asserts heat and light to be modes of motion, is at present obscured by two clouds. I. The first came into existence with the undulatory theory of light, and was dealt with by Fresnel and Dr. Thomas Young; it involved the question, how could the earth move through an elastic solid, such as essentially is the luminiferous ether? II. The second is the Maxwell–Boltzmann doctrine regarding the partition of energy.[0]

As I understand it, the two clouds were dealt with via special relativity and quantum mechanics.

[0] https://books.google.com/books?id=YvoAAAAAYAAJ&pg=PA363#v=on...

Our society has changed, dramatically, in the past 30 years. We've gone from a completely disconnected society with connections spanning arm's reach to a globally connected society with connections spanning just as far. Baby sitters have been replaced with digital tablets providing endless entertainment and ensuring no person, adult or child, ever need sit with their own recreation being solely within their mind. One needs only look at the slew of digital devices whipped out at any eatery, lest individuals be left to their own thoughts for an agonizing 10 minutes.

The internet and our technology has shattered nearly all educational barriers. Getting an MIT caliber education for free is now something any child in a developing country who has access to the internet can do, if so inclined. On the other hand they can find countless entertaining drama to read or participate in, find an endless supply of porn, engage in social media and other online sites that are specifically designed for addiction, or find an array of games and other forms of fast response stimuli all designed to make him or her feel good.

Einstein constantly related his tales of meandering strolls as he pondered his work. Another individual that has managed to achieve some great things in modern science is Stephen Hawking. He's mentioned spending decades thinking intently, even if not exclusively, on singular topics. And there is absolutely every reason to believe that is in no way bluster. Given his condition, his mind is his recreation.

Is the breaking down of information barriers in our society helping more than the massive commercialization and directed addiction towards all sorts of mediums and vices along with the rapid reward systems they offer, is hurting it? It's not just high level academia that's failing. Education systems to be deteriorating at all levels with US (which is the center of much of this change) scores in math and science rapidly falling behind much of the rest of the world. Healthfulness has fallen to dangerously low levels. And more. Regression is not impossible.

Although, to be fair, the direct effect is explained by increased inequality:

http://www.eera-ecer.de/ecer-programmes/conference/20/contri...

Yet, I know how digital shit affects me. I'm not talking about the devices themselves, I'm talking about being flung up by some stupid pointless shit in social media or by the colorfull game which totally hijacks the reward mechanisms in my brain.

I totally fail to see how the effects which promote shallow, fast reward seeking cognition are not not detrimental... yeah yeah, it sounds like greeks whining about reading spoiling the mind but the difference is that our media is engineered to distact and hijack the mind for no valuable outcome whatsoever.

Sure, there are lots of positive effects but that does not mean one should not discuss the negative ones as well.

I'm currently writing my phd project (after having worked for 10 years in a company) and I'm absolutely stunned by the current easiness to access papers and informations compared to just 10 year ago when I left university.

I concure however that a new and real challenge is now to be able to filter the noise induced by this overwhelming volume of available information. But I don't think nostalgia is an useful tool to reach this goal.

PS: To illustrate https://xkcd.com/1601/

(And I believe there was another comic that it traced it back to Greek philosophers that rejected writing. Thanks to whoever remember the link!)

I wouldn't generalize this to all branches of science. My own experience in physics contradict that, and I also get the impression that some of the large successes we've seen recently (like gravitational wave detections) couldn't have happened if the scenario you described were the norm.

My advisor and I have found some questionable stuff in PoP, for sure.

"Well it was the engineers who did most of the work..."

Who do you think came up with some of those engineering advances? No need to play semantic scientist vs. engineer arguments. Some of the advances came from scientists and some came from engineers.

But gravitational wave detection wasn't "just" engineering; there's a lot of theory to be done to figure out what signals to look for, how to interpret them etc.

The original post here is talking about reproducible results and p-values and publishing. Is LIGO a particularly good example of that part of science being done well? I don't know either way.

FWIW I'm much closer to being a scientist than an engineer.

I don't see how you can look for a specific kind of outcome from an experiment without fooling yourself, or cheating.

It's much like with tech startups - you can design a startup such that it will either be worth billions or zero by tackling a problem that most people believe to be unsolvable but will affect lots of people, and then trying multiple counterintuitive ways of solving it. The vast majority of them will fail because that's what it means to be counterintuitive, but if it succeeds, you have a world-changing company.

Financial instruments as well - an option is a financial contract whose value will either be zero or a lot, depending on whether an unlikely-but-possible event occurs.

Not nearly enough of them of course.

But still, it's kind of funny how certain fields in pseudoscience actually drive progress in statistics and scientific methodology :)

He even made a simple problem more complex just to fit his base theory. It's only about getting citations.

You can look at any experiment as having essentially two possible outcomes - you need to ensure that at least one of those outcomes is publishable, because there are many experiments where any possible outcome will be weak.

Gelman's other point is this is all made much worse in inherently low-power fields like psychology and the social sciences. A field like physics can "get away with" it more because making the appropriate corrections is less likely to wildly change the conclusions.

[1]: http://www.stat.columbia.edu/~gelman/research/unpublished/p_...

This is the advice of a true scientist. I'm refreshed to see that they're still out there.

It encourages students to cheat, because most of the time you won't get either. Not a significant result, and usually not a counterintuitive one either. You're pretty likely to get a significant result that agrees with existing theory, though.

> The most widely cited test was a 1987 study for Bicycling magazine by engineering professor Chester Kyle, one of the pioneers of cycling aerodynamics. He found that leg-shaving reduced drag by 0.6 per cent, enough to save about 5 seconds over the course of one hour at the brisk speed of 37 kilometres per hour. At slower speeds, the savings would be less.

> [More recent tests in a modern windtulle show] that [shaving legs reduces drag] by about 7 per cent (...). In theory, that translates to a 79-second advantage over a 40-kilometre time trial that takes about one hour.

> [The aerodynamicists in charge of the windtunnel contacted Kyle], to ask if he had any ideas about the discrepancy between the two results. It turned out that the 1987 test involved a fake lower leg in a miniature wind tunnel with or without hair glued onto it – hardly a definitive test, and yet it was enough to persuade most people not to bother with further tests for the next three decades.

https://beta.theglobeandmail.com/life/health-and-fitness/hea...

Edit:

It's curious to me, because in other areas it seems some degree of surface detail reduces drag (golf balls, aeroplanes too I've heard) - I wonder if short stubble is actually better and that's the effect you get using real legs. That would account for a plastic model only having a small gain.

Assuming that we're already reaching the tail-end of the bell curve for human performance in cycling sport (which seems likely to me), suggesting minimal individual performance differences, then being at among that group despite a 7% performance disadvantage seems downright impossible.

So especially at the top there would be an incredible selection against anyone with hairy legs.

But I guess you could say that natural selection beats human belief.

I'm concerned about it from a different angle:

How much public policy, medical treatment, and follow-on science is seriously suboptimal due to this house of cards?

This you do not want to know.

Here is a good question, the politics and bad science of which were uncovered by the documentary "Bigger, Faster, Stronger: The Side Effects of Being American" - why are exogenous testosterone and anabolic steroids strictly controlled and essentially banned?

What is your first reaction to this question? That they are unhealthy, that they have gross and terrible side effects? Likely this is what you have heard. Nearly 100% of what is popularly "known" about anabolic steroids and use of testosterone is absolutely false.

Exogenous testosterone was banned because a father of a boy who committed suicide (who was taking steroids [0]) rode a publicity wave all the way to Congress, despite the recommendation against the ban by the AMA and other medical experts.

Now, think of what else this may be true of?

[0]: The son was taking steroids, yes. He was also on benzos, drinking alcohol, and doing other recreational drugs.

Correct me if I'm wrong, but I thought the main issue was that papers were being replicated, but the results weren't being published (especially if they were negative). These problems were always being discovered, but the record was never being corrected.

You may be right, but that's basically the same thing (and as you have sort of pointed out, it's not possible to tell either way).

If you just do the experiment again that's repetition.

So experiments are being repeated, but results can't be replicated and the results aren't being published, as you note.

The unpublished results perhaps aren't strong enough to make a refutation?

1. Replicability of social psychology results are estimated at 25%. (Cited in article, from http://www.nature.com/news/over-half-of-psychology-studies-f...)

2. People are not reporting non-significant results. Which means we get an incomplete picture of the validity of certain hypotheses. (from article)

3. The numbers don't line up when comparing observed power and significance of results. (from article)

4. Some results data is erroneous or falsified. The GRIM test that surfaced a while back shows numeric "results" that cannot possibly be derived from the sample data. (https://medium.com/@jamesheathers/the-grim-test-a-method-for...) (https://www.vox.com/science-and-health/2016/9/30/13077658/st...)

5. Many authors on the papers don't want to share the data from their papers - even if it's a condition of publication. (from a previous article about the GRIM test that I can't find) This doesn't directly equate to maliciousness or deceit, but it illustrates incorrect attitudes in the field. Some authors consider it undignified to have someone "check their work". Others are concerned of fallout if their data contains errors. (Generic mainstream blog/press article on reproducibility https://www.vox.com/2016/3/14/11219446/psychology-replicatio...)

6. Ridiculously small sample sizes are often used in experiments. Often this doesn't seem to affect how the results are received, though it should. (briefly discussed in article and comments)

7. Papers and studies are cited in articles and future papers before they've been peer reviewed, replicated, or proven sufficiently to justify their inclusion into the field. Thus the results are "baked in" to social psychology as fact at an uncomfortably early stage. (http://blogs.plos.org/mindthebrain/2015/12/30/should-have-se...) This kind of feeds into the hype machine where people focus on significant results - even if the data needs massaging to get there. Even after being refuted (if refuted) the field and its fans never seem to fully recover and back away from bad science. By then it's already made its rounds on Facebook and it's even referenced in the worst of the college intro to psych/sociology courses.

8. Journals in general need to be more open with their data. Especially with publicly-funded research. Paywalls don't help progress.

9. Corporate/politically funded white papers publish studies without making the connections clear.

The field needs a massive overhaul in the sense that its participants need to stop chasing controversial results in the press and instead focus on reproducible results and methods. All fields need some of this, but psychology needs it more than others.

That's a pretty big reason not to publish a negative result, especially if the original result has social credence (big name, popular publication, wide acceptance).

I would actually expect the opposite, as finding evidence to support a new claim should be more difficult than not finding evidence.

My understanding is that a negative result has the following format: "we did such and such experiment and did not find a significant statistical relationship between thing 1 and thing 2, after controlling for a bunch of other things".

It's worth noting that this by no means proves that there isn't a relationship, it just means that study wasn't able to find evidence of one. It could be a piece of the puzzle for a potential strong case against such a relationship, or that further research is needed to untangle any confounding factors. Which is why I think all methodologically sounds results should be published, no matter how unflashy or boring.

(And yes, it is my opinion that climate change is absolutely happening. Does not mean that we can't be skeptical of the effect sizes cited.)

"And yes, it is my opinion that climate change is absolutely happening" .

The level of fear in stating simple opinions has gone to 11.

No possible discussion follows. All opinions, however tame, are outcast.

Or is this the point you are making, and you are expressing a climate-change-denying position, really obtusely? If so, it's on you to show climate science is 'spinning tenuous research into facts'.

Reduce that to:

> happens to lead us to a conclusion we don't like, we should be careful especially given the immense political pressure both within science and in governments.

Liking has little to do with it.

Those who are purists also don't like that others question dogma.

Right now, submitting data that contradicts a published paper is likely to be ignored, treated with hostility, or swept under the rug. At very best, you'll get a small publication of a retraction.

Disproving published results needs to be at least as prestigious as producing novel results.

If we can't swing that - it would be great for graduate student research work...

I don't know if it's true, but it sounds like it might be a good idea.

How might we get funding though? Perhaps with a prediction market (people bet on whether scientific hypotheses will be disproved/confirmed)? It could be fun.

I'd love to prove things wrong for my master's thesis but novelty is a grading criterion so that's a no-go.

This kind of stuff despite those flaws gets propagated without much skepticism. It makes me wonder how much stuff we "know" to be true is the result of an intuitive argument but has no real support.

It is scary, and you are right, but you can do your part by blogging about it. Doesn't have to be official, just post the text and your notes. The more people do this, the more we have informal peer-review and replication checking after the fact, and at least a loosely-connected database of blogs and articles will exist when people search the article title.

It's fascinating to see the "new guard" emerging in psychology, actively acknowledging the poor practices that have preceded them and fighting back were possible.

A while ago I had a manuscript rejected because I used the word "novel"... so hopefully the trend of emphasizing "novel" things [1] is changing.

I'm trying to find the specific manuscript guidelines for that journal, but pretty sure they banned use of the word.

[1] http://www.nature.com/news/novel-amazing-innovative-positive...

The criteria for publication of scientific papers (Articles and Letters) in Nature are that they:

-report original scientific research (the main results and conclusions must not have been published or submitted elsewhere)

-are of outstanding scientific importance

-reach a conclusion of interest to an interdisciplinary readership.

"Outstanding scientific importance" screams no boring replication studies that confirm pre-existing work.

The note in the first bullet also forbids replication studies that confirm pre-existing work.

> (the main results and conclusions must not have been published or submitted elsewhere)

I don't know, "couldn't replicate this other major study" seems like it would qualify as outstanding scientific importance. It all depends how you frame "scientific importance". Replication seems like it should be high on the list of scientific goals.

You're missing the other side of it. Here's what's not interesting: Replicating the results of a study already believed to be accurate. Yet it is vitally important.

To arrive at the simplest truth, as Newton knew and practiced, requires years of contemplation. Not activity. Not reasoning. Not calculating. Not busy behaviour of any kind. Not reading. Not talking. Not making an effort. Not thinking. Simply bearing in mind what it is one needs to know

It would also be interesting to repeat the experiment with studies from different points in time (when there was more ideological diversity) to see how the effect varies with homogeneity. My guess is that if there is an effect, it varies nonlinearly with diversity--in other words, a field with 90% political homogeneity might have an effect that is ten times that of the same field when it was only 70% homogeneous. I suspect that once you hit 90%, dissent is much more effectively suppressed than at 70%. But maybe not?

The academics will deny these hidden truths but the greater population will (and already do) exploit them.

http://www.joseduarte.com/blog/wheres-joe

I can attest there is a lot of crap science published. So much so that when you find a really good publication you think "wow!!"

The one thing about industry is that if the science doesn't work you don't make money. However there is little incentive to publish.

And worse, the vast majority of the evidence cited in major social science debates is deficient in this way.

Secondly, supporting OA-friendly journals is the next step. PeerJ is a very good initiative on that front for a lot of reasons - they open and document the peer-review process, which is unheard of. It's a tremendous asset to look through peer review and see how the "science" is really done. I highly recommend everyone find some papers that they are interested in on PeerJ and check the peer review logs.

I don't think it's any one thing or that there's an easy fix. For my part, I refuse to publish in non-OA journals and strongly prefer those who require/strongly encourage both open data and open peer review. PeerJ is one such journal which also features very fast time to first decision AND low fees. I cannot recommend them enough.

Replicating research would boost your reliability score and the original's too which would boost credibility for future research.

Are there any departments or disciplines or academic careers that can be made on scrupulous scientific study of how science goes right and wrong, including statistical as well as other issues? I think we need it.

https://www.wired.com/story/why-men-dont-believe-the-data-on...