Filling in the cracks of doubt is a big part of what replication and productionizing is for. Of you think a study has a big enough hole in it's methodology, you are free to go design your own study to try to plug the hole or refute the result.
A corollary, though, is that there is always going to be a 'hole' big enough for a jackass on the internet to criticize. But the question is whether you're willing to put the skin in the game to see if the result holds when you design a study to exploit the hole.
Even in mathematics ... others on HN are better qualified than I am to comment, but I've read mathematicians say that almost no reader of a paper works out all the math; to a degree they go by reputation of the author that the mathematics is generally correct.
Similarly to your comment, while outsiders envision perfection, people inside have limited time for it. It may be more productive to spend that time on new research than on investigating every little detail of old research.
We call it science because when we find methods that we all agree on, as a culture to use then it becomes scientific.
For Instance psychology, in the freudean psychology the methology is still science even though he speaks of urges of "your mother" and interesting things like the "subconciousness". It has been met with great doubt over it's "initiation" but is now a widely acceptable practise.
So what has changed? Well the field concentrated on scientific core value which (almost) boil down to: Find a methology, prove that it works and replicated it again.scientific quality criteria.
I agree with op! Stimulating cells with electricity is not teaching them. Its assisted muscle reaction.
In this case its really clear cut too: if I need to give the braincells electro schocks (the thing they operate on), If anything, I am remote controlling brain cells. The whole idea of perception is flawed in this experiment, you know, because there is no actual perceptor of the paddel to begin with.
Sorry but as someone with a bachelor in psychology and 20 years of dev experience the starting poing of "I think many non scientists have a bit of a wrong idea of how science actually works." is just condecending theory. This is literally what "scientific quality criteria" is based on.
Psychoanalysis is only the tip of the freudian journey and i was not trying to say that everything he did on a cocain run was perfect or even valid.
What i was trying to say is that the scientific process for this has evolved and in reality what diffencietes science from pseudo science is methodology. Psychology is literally the method of the psyche (translated).
Even this mythbusters says: the difference between science and screwing around is to write it all down ;)
Also, most results that reach the front page here are groundbreaking. But groundbreaking results are very rare, so most of them must be fake, because they have an error or because they are an exaggeration.
Most good results are too technical and boring. My favorite example is Magnetoresistance. It's a weird quantum effect that is caused because electrons can have spin up or down. It has a huge effect in the applications and almost everyone had a device that used it. Now the applications are fading. But no one knows it.
There are probably 1000 times more discussions here about faster than light travel than magnetoresistance.
While preprints lack some work and signals of reliability, such as peer review, revisions by the author, etc., I think they are far beyond blog posts. Imagine a blog post, by an expert in the domain, with the amount of research and precision that goes into a preprint - there's no comparison.
Nothing, not even a peer-reviewed article in Nature, is so reliable that we can just put away our critical thinking.
Incentivizes what? What is hard to resist?
I’ve seen this happen a few times in papers that were never fixed, nor the issue brought up as far as I’m aware. Granted, in each case the thesis was still fine but… makes me wonder about the cases I don’t know about. One of the cases skirted really close to the edge of bs.
1. Science is hard.
2. There is no such thing as a perfect study. Every method/methodology has weaknesses. Look for converging evidence across multiple studies which use different designs.
3. Initial studies often do not have the funding needed to do things as optimally as possible in a single study, but they do provide evidence for ideas that can be pursued further.
4. It is often socially safer to point out deficiencies than to praise strengths. Same as snobs raining down on you when you say you like some midrange wine.
5. Often issues will be raised that are already acknowledged by the authors.
6. Complaints are often based on inaccurate news reporting, not actual research.
7. Science is hard.