News for hackers. Stuff that matters.
Re:A new replication crisis: Research that is less likely to... by kqr on Saturday May 22, 2021 @05:13AM (#27244891)
When I was in school pre-university, this type of "crap we can't get the what we wanted to happen so let's just fiddle around with it until it seems about right" was very common. I was convinced this was how children learned, so that as adults they wouldn't have to do things that way.

When I got into university and started alternating studying and work, I realised just how incredibly clueless even adults are. The "let's just try something and hope nothing bad happens" attitude permeates everything.

It's really a miracle the civilisation works as well as it does.

The upshot is that if something seems stupid, it probably is and can be improved.

Re:A new replication crisis: Research that is less likely to... by patrakov on Saturday May 22, 2021 @06:42AM (#27245274)
In the lyceum where I studied, there was one lab on Physics, where the book that accompanied the lab was deliberately wrong. We were told to perform an experiment that "should" support a certain conclusion, but actually neither the "correct" conclusion nor its opposite could be done because of the flawed setup which measured something slightly different. A lot of students (in some groups, all students) fell into this trap and submitted paperwork with the "correct" conclusion according to the book.
Re:A new replication crisis: Research that is less likely to... by brandmeyer on Saturday May 22, 2021 @11:14AM (#27247304)
A CS-specific analogy might be to give the students a compiler that has a bug in it, such that the students' code is deliberately mis-compiled. The standard of evidence to believe that the compiler is buggy is much higher than the standard to believe that my code is buggy.

A lab exercise like that could really just be selecting for chutzpah (feeling charitable) or arrogance (less charitable).

Re:A new replication crisis: Research that is less likely to... by patrakov on Saturday May 22, 2021 @03:09PM (#27249410)
Well, that's more evil than my lab. A more direct equivalent in the CS would be an algorithm description in the booklet with one subtly wrong (e.g. proven using some well-hidden circular reasoning) and uncorrectable step. The expectation would be that a good student finds the mistake instead of submitting the implementation of the flawed algorithm, or, for even better matching with my case, proves that the supposed output cannot be obtained from the inputs at all.
Re:A new replication crisis: Research that is less likely to... by tonyarkles on Saturday May 22, 2021 @10:00AM (#27246609)
I had an appendectomy just before the final first-year Modern Physics lab and had to come back in to do a make-up lab. Sure enough it was the slightly-messed-up lab where the results should in theory look exponential but come out linear. I, naturally, drew an exponential curve through the points. Lab instructor decided to grade it right there before I left and tore a strip off me.

Very valuable lesson, although it sure did suck at the time.

Re:A new replication crisis: Research that is less likely to... by economusty on Saturday May 22, 2021 @10:31AM (#27246871)
What does tore a strip off me mean?
Re:A new replication crisis: Research that is less likely to... by kencausey on Saturday May 22, 2021 @10:45AM (#27247007)
Re:A new replication crisis: Research that is less likely to... by bussierem on Saturday May 22, 2021 @04:24PM (#27249960)
And for those ALSO thinking "what does THAT mean":

He got criticized for it.

Re:A new replication crisis: Research that is less likely to... by vidarh on Saturday May 22, 2021 @06:25AM (#27245200)
I've come to think things works as well as it does largely because a whole lot of what people do has no effect either way. I see so much stupidity where the only saving grace is that it is directed into pointless efforts that won't be allowed to do any real damage.
Re:A new replication crisis: Research that is less likely to... by Retric on Saturday May 22, 2021 @08:35AM (#27245908)
When you start talking millions of people damage gets subtle.

Robocall scams are very high on the profit:human misery scale, but their hardly going to end civilization. Pollution, corruption, theft etc all make things worse, but we never see the better world without such things so it all feels very abstract. Of course you need to lock your doors etc that’s just the way things are.

Re:A new replication crisis: Research that is less likely to... by api on Saturday May 22, 2021 @08:11AM (#27245746)
A bio professor of mine said something that stuck with me: “life doesn’t work perfectly, it just works.”

It has to work well enough to… work… and reproduce. That’s it. It’s not “survival of the fittest.” It’s “survival of a randomized subset of the fit.”

There’s even a set of thermodynamic arguments to the effect that systems are unlikely to exceed such minimum requirements for a given threshold. For example, if we are visited by interstellar travelers they are likely to be the absolute dumbest and most dysfunctional possible examples of beings capable of interstellar travel since anything more is a less likely thermodynamic state.

So much for Star Trek toga wearing utopian aliens.

Re:A new replication crisis: Research that is less likely to... by cutemonster on Sunday May 23, 2021 @05:31AM (#27253639)
> if we are visited by interstellar travelers they are likely to be the absolute dumbest and most dysfunctional possible examples of beings capable of interstellar travel

Otoh, they would be aware about that, and they might have spent some time improving how genes (or what they have) and evolutionary selection works for them, so that, say, their species with time becomes brighter and brighter than what's actually needed. If they wanted to do that.

Re:A new replication crisis: Research that is less likely to... by api on Sunday May 23, 2021 @10:05AM (#27255074)
How do you improve your genes? Removing obvious deleterious disease mutations is easy, but as soon as you try to “go where there are no roads” you hit the same combinatorial challenge as evolution.

Also more intelligence does not equal better ideas. The world is full of crazy or amoral people with apparently very high IQs. Your average flat Earther probably has an above average IQ.

Improvement is a war against entropy and n^n^n^… combinatorics any way you slice it.

Re:A new replication crisis: Research that is less likely to... by cutemonster on Sunday May 23, 2021 @10:53AM (#27255381)
> How do you improve your genes?

Slowly across hundreds and thousands of generations.

By adding evolutionary pressure, for what they want -- it'd be up to those space traveling aliens to decide -- they can change their species, generations into the future.

> Improvement is a war against entropy ...

Reasoning in that way, the humans would not have gotten brighter than the chimpanzee monkeys. There's been evolutionary pressure for the humans to get brighter, and it would be possible for you (I mean the humans), or the space travelers, to add artificial ev. pressure.

Anyway never mind all this, maybe talking about space travelers and the humans and their genes isn't the best way to spend the day. Have a nice day btw

Re:A new replication crisis: Research that is less likely to... by agent008t on Saturday May 22, 2021 @06:29AM (#27245219)
The problem is the incentives. To do well, you must publish. To publish, you must have a good story, and ‘we tried this and it didn’t work’ is not one.

So after a certain time spent, you are left with a choice of ‘massaging’ the data to get some results, or not and getting left behind those that do or were luckier in their research.

Re:A new replication crisis: Research that is less likely to... by hef19898 on Saturday May 22, 2021 @07:51AM (#27245621)
"We tried this and it didn't work, and here's why we think it didn't" should be among the bests stories to publish. Looking back I learned more from stuff that didn't work, or rather figuring out why it didn't, than from success.
Re:A new replication crisis: Research that is less likely to... by dagw on Saturday May 22, 2021 @08:34AM (#27245900)
or rather figuring out why it didn't

That can end up being just as time consuming as doing the research to begin with. Often there is no time and no money to go back and do that. If your 'budget' is 6 month you're going to spend 6 month trying to get your experiment to work. You're not going to 'give up' after 4 month and spend 2 month putting together a "why we failed" paper.

Re:A new replication crisis: Research that is less likely to... by bumby on Sunday May 30, 2021 @03:29PM (#27335917)
However, the advantage is, if it is published, it can decrease the likelihood of multiple other attempts to try the same “unique” (and wrong) approach.
Re:A new replication crisis: Research that is less likely to... by j7ake on Saturday May 22, 2021 @08:20AM (#27245809)
Even if something did not work, you still need a story for it to be readable.

For example, I imagine that archeological work is extremely high impact if excavation efforts led to discovery of ancient city.

Archeology paper would probably be less interesting if the paper said “we dug this area, found nothing”.

If one were to judge those two papers, obviously the discovery paper is higher impact than the negative result.

Re:A new replication crisis: Research that is less likely to... by dtech on Saturday May 22, 2021 @12:21PM (#27247982)
"We chose this area because we believed it should be archeologically interesting based on XYZ. However, we searched through ABC methods and found nothing there" would be valuable for the future. Maybe XYZ isn't as good as we though, maybe ABC couldn't find it. Maybe now some other sod in the future won't try that location.

Not as valuable as a discovery, but very far off zero value. Yet the reward in academia would be near-zero.

Re:A new replication crisis: Research that is less likely to... by Frost1x on Saturday May 22, 2021 @08:16AM (#27245784)
Tell that to the people who pay for research and the metrics they use to continue feeding those who perform research.

Ultimately, this is the problem.

Re:A new replication crisis: Research that is less likely to... by katzgrau on Saturday May 22, 2021 @08:26AM (#27245849)
Distracting from the main point, "let's just try something and hope nothing bad happens" (trial and error) is precisely the reason civilization made it this far :)
Re:A new replication crisis: Research that is less likely to... by simonh on Saturday May 22, 2021 @11:45AM (#27247593)
And in fact evolution. The thing to remember is, in many cases where something bad did happen the evidence got buried or eaten.
Re:A new replication crisis: Research that is less likely to... by tambeb on Saturday May 22, 2021 @04:12PM (#27249893)
> It's really a miracle the civilisation works as well as it does.

I think this all the time.