Once, years ago, I got into a debate with somebody online who just kept dropping links to studies after making a statement as if it proved them correct.

One day, I got bored and decided to actually read every single study. The studies said nothing close to what the debater suggested and seriously opened my eyes to just how terrible some studies can be in terms of quality.

Now, I just assume that anybody who starts link bombing in a conversation has no idea what they're talking about and can't engage on a logical discussion.

If you ever want to reject a study, just read it.

You will find something to nit pick.

I'm not saying you're wrong — I've got my name on exactly one paper so what would I know — but that being true would suggest that peer review is fundamentally broken.

No, it would suggest that every paper has something in it that can be considered a fatal flaw by a person biased against it. Peer review is precisely supposed to admit this and minimize flaws in conclusions given inevitable flaws in methodology.

Sort of a research paper equivalent of:

"If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him." by Cardinal Richelieu

Peer review is not a certificate of truthfulness, peer review just says that a grad student reading the paper did not found anything too suspicious there and it looks like a typical paper in that field.

Peer review, at its core, is a social consensus process. It can work, but it is structurally inclined to propagate agreement.

I wonder if it serves to reinforce this problem, as those more motivated to reinforce certain literatures or perspectives will surface those repeatedly in reviews.

There's a popular work that comes to this conclusion in its beginning chapters [0]. The basic premise is that the act of science and knowledge acquisition cyclically devolves into ideology and is then disrupted. Typically those who initially disrupt a scientific dogma are not treated well. Eventually the old guard literally dies off and the new ideas can begin to take hold.

[0] The Structure of Scientific Revolutions, T. Kuhn

> The basic premise is that the act of science and knowledge acquisition cyclically devolves into ideology and is then disrupted.

This can happen of course, but it's not really what Kuhn is talking about. Kuhnian paradigm shifts are to science what massive refactorings are to a codebase: the problem isn't that the old code was wrong (though it might also be wrong), it's that it can't be extended to meet new requirements. And while some of us are better at writing extensible code than others, no one gets it right every single time.

A huge part of the problem is that the layperson is taught otherwise.

2nd grade we learned the scientific method. I don't remember peer reviewed being a thing.

Replication was critical.

Also, I've seen peer reviewed papers with atrocious stuff in them, unless I know the peer, it doesn't mean much to me. Give me 100 independent groups coming to the same conclusions. That has far more weight than a few of your friends/professors reading your paper.

Not even that. I was a grad student that reviewed a paper submitted to a well-known journal from a heavyweight in my field. The paper's own data showed that the authors were reporting 95% noise. My advisor rejected the paper only to see it published a few months later (in another famous journal) after the authors removed the data that allowed us to detect the noise throughout the data.

Not exactly. Control for that in a future experiment?

To be fair, as you start to leave chemistry, true knowledge(the goal of science) is impossible to find. The best we can get is little glimpses of the truth.

To be fair, I have seen studies that were pretty bullet proof. It just seems that most are lazy or simply impossible to prevent variables.

I mean I think peer review kind of is fundamentally broken, but at the same time this phenomenon isn't necessarily a sign of it.

Even within technical papers, let alone pop writing or internet discussions, many of the claims citations are provided for are broad e.g. "X can increase people's anxiety". A very rigorous study might exist showing that when X happens a specific way it does increase some form of anxiety in some specific subset of the population, but using it as a citation for the broad statement without further context can be misleading at times.

It's in fact possible that by considering different subsets of the same statement you'd get an opposite directional effect. That can certainly be used to confuse people, especially in a situation where it is unlikely most readers will dive deeply into the cited works.

As far as nit picking - the same general principle applies. There will always be some tradeoff between scope, rigor, and available resources to do the study. If we waited for papers to be perfect in both correctness and interestingness/utility there would be almost nothing to ever publish.

So I think the systemic problem here is moreso an undervaluing of review article and text book type resources (both reading them and writing them) in favor of vomiting out random individual paper citations for whatever claim. Science needs more heterogeneity in the roles different PIs fill for the system.

Improving peer review process would be great (and might indirectly help), but I don't feel it's the root.

When reading the first draft of the California Math Framework, I followed many of the citations. My experience was similar to yours. In particular:

- some of the cited studies did not claim what the citation said they did

- some of the cited studies did claim what the citation said they did, but the experimental results were far too weak to support these claims

- some of the cited studies had such weak experimental design, that it would be hard to conclude anything about the subject at hand

This is one reason Wikipedia discourages using primary sources (i.e. individual studies) as references. Not that using secondary sources is perfect either, but you can "prove" almost anything by linking to studies as there are just so many of them. Do 1,000 studies on homeopathy and some of them will show a positive effect (even when done well, ignoring there are also many bad studies on these kind of topics).

> One day, I got bored and decided to actually read every single study.

Nothing is more entertaining than the meeting after a business presentation where someone fact checks the reference after the power point. When we have vendor presentations, we started doing this because... so many claims, so little truth.

Yep. This is why it makes me nearly physically ill when I hear people use terms like "science denier" or say things like "<insert political party here> don't believe in science" or "our laws should be based on science".

It is invariably some of the most scientifically illiterate, ideologically entrenched, and intellectually lukewarm people who spout this garbage as a retort to any sort of argument with which they do not wish to engage.

My favorite term like that was “scientific consensus”. I used to point out that up until the 17th century the scientific consensus was that the Sun revolved around the Earth. I would get nothing but blank stares in response. It turns out that people who talk about “scientific consensus” typically don’t know whether the Earth revolves around the Sun, or vice-versa.

It is especially frustrating when scientific ideas are applied to moral dilemmas. I see this from the far left wing frequently. The irony is that this same mindset is what led to the proliferation of eugenics apologists in the 1800s. The scientific consensus had absolutely no objections to eugenics on scientific grounds. In fact, "science" would seem to support it. Yet it is widely agreed that this practice is morally reprehensible nowadays, because humans were able to put aside their hubris and apply their moral reasoning.

Plenty of people realized that eugenics was unacceptable who were in the scientific world then, just like plenty of people understood that slavery was wrong. Which doesn't take away from the fact that plenty of terrible policies came out because leaders did have those views. I don't know what left wing ideas you've seen that are wrong, I'm sure that I've seen some left wing ideas that are wrong. I'm more worried about what do the leaders and the effective speakers in groups promote. Saying that I'm making you more free by taking away your choice of library books, or your choice about medical care, that's not an extreme view that only 1% of people have. It's pretty much the consensus leadership view of Republicans.

I'm staring blankly too. The idea is to forever invalidate consensus because people were wrong in the past?

Consensus means that people agree; it doesn’t mean that they are right.

Yeah, but it means they're far more likely to be correct. Iconoclast / Galileo gambit outliers are near nil.

We collectively know more about everything every year, science has generally given us a self-correcting living corpus.

The argument you're making is the same made by anti-intellectuals: don't trust institutional knowledge and consensus because it's been wrong in the past, look at mistake X. It's something like a systemic ad hominem.

> Yeah, but it means they're far more likely to be correct

I know what this means from a statistical perspective but I dont know what it means with regard to practical day to day decision making.

> The argument you're making is the same made by anti-intellectuals Isn't there some sort of fallacy that you are yourself deploying here?

The fallacy is assuming that consensus = probably true.

This is only correct when we have good evidence and good methods. That is why consensus breaks (like heliocentrism) occur during technological changes (development of telescope).

The critical point the parent is missing is that many fields have terrible data and terrible methods so the ‘consensus’ isn’t likely to be true (eg smoking is good for the nerves, butter is bad for you, the globe is too big for humans to cause changes)

It takes a lot of intellectual integrity to admit that many disciplines are not generating anything resembling ‘truth’. It’s easier to ignore and say: but the consensus is the best we have! Sometimes the heterodox view is the best.

They are not near nil. Every idea ever conceived has either been rejected or altered to match new observations.

That means that most ideas will end up being wrong in some way. The smoking-gun laws of nature are few and far between.

If you consider what % of ideas throughout history were plain wrong, it’s hard to believe that we have finally figured it all out and the consensus finally reflects reality most of the time.

We're talking about wrong as in Galileo, as in radically upending, not modifications or normal gradients of wrongness. Knowledge is generally accretive, but we pay a lot of emotional and categorical attention to perspectives that break dogma. The vast majority absolutely do not, and they fade innumerably into the background of the industry they support.

It's easy (and trite) to cite the extremely limited pool of successful dogma-contrarians over time. Try making a list of the failed contrarians, and then another of all the knowledge (and its contributors) that didn't need fundamental reconsideration at some point in modernity.

The critical point is that contrarians are always necessary for science. If we always ignored dissenters then we would never make any progress. Even if most are wrong, contrarians are essential.

Thus deferring to the ‘consensus’ and implicitly ignoring contrarian views stops science in its tracks.

I don’t understand what you mean in your first paragraph, and I don’t support dogmatic positions so I’m not sure why you refer to ‘dogma-contarians.’ Contrarians are usually anti-dogmatic.

“The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man.” Unreasonable contrarians are generally annoying but essential.

The tricky bit is there really are people who think that science "is just an opinion, man". See e.g. "teach the controversy" regarding evolution vs. creationism. There certainly a place for "<X> don't believe in science" and "our laws should be based on science".

But the argument is so often misapplied that it's become meaningless.

I have argued against 5G deployment for example, not based on outlandish "zomg Bill Gates George Soros mindcontrol!!!11" or "brain tumours!!!11" nonsense, all of that is clearly nonsense. But the science is a lot less clear that there are zero effects than is sometimes made out to be, and there are also the ethical considerations of informed consent. I was, of course, immediately lobbed in with the crazies and called a science-denying conspiracy theorist, by someone with no expertise in the field who said I need to "listen to the science", in spite of my argument looking nothing like that.

Don't even get me started on COVID – any attempt to inject even the slightest sort of nuance was met with "you are literally murdering people with your unscientific nonsense!" and you were immediately lobbed in with COVID-denying anti-maskers or whatnot. At some point this stopped being a debate about trade-offs involving science and medicine, and basic liberties and freedoms on the other and became some sort of moral crusade (and it seems it still is; there was a conference this month where masks at all times, full proof of vaccination, and a PCR test was still required, which seems a bit much for 2023).

A big issue is that any sort of nuance is often met with the most uncharitable interpretation because the genuinely crazy people have been getting so much attention.

The thing is, based on so many examples of people who cite “the science” that are clearly exaggerated or unsupported “just an opinion” isn’t too far off.

It shouldn’t be this way, but there’s a lot of undermining of trust in science because of this stuff.

Take evolution as an example. How many layers of scientific expertise have to be understood to really claim that you understand how evolution works? Archeology, biology, history, geology, genetics…potentially more?

At some point there’s a trust factor involved in accepting evolution.

Now apply that same realization with climate change.

The more complex, the more moving parts, the easier it is to find a part to be skeptical about and people will do exactly that. Especially if they are given reason to believe that the science is just there to support a political objective.

In the end, unless science can be easily replicated and demonstrated (gravity, boiling water, killing bacteria, generating power, flywheels, etc) it will boil down to trust for the vast majority of people.

I've come to the conclusion that science needs to be just another form of religion (without the theistic element). I don't have the time or the energy to go through the research to determine if climate change is real. I put my faith in the scientific process, which is probably the most successful thing we've ever come up with as a species. I'm not sure why I should believe some random dude, whoever he might be or what credentials he might have, on the radio/TV over the scientific community. Sure, they've gotten things wrong, but their success rate and their usefulness to our species is infinitely better than some politician being a climate change denialist just to appeal to voters. What process does he have and why should I trust it more than the scientific method?

I disagree with that attitude. I think most scientific information that comes to us through the media is reliable. If people stop trusting science (or they continue to stop trusting science), we are just left with superstition and religion. It was really hard for people to cope with the fact that during covid we started with the best explanations and then as we learned we improved on it and some ideas or expected safety practices changed. Good example is that many viruses spread through touch and covid spread through the air and that was a surprise.

People take that in and say I just don't trust anything. That is a problem with America because people stop believing in objective facts, saying that it goes against their beliefs. This is a major problem why America doesn't have enough engineers and scientists and mathematicians, because people haven't learned that your intuition can be wrong and you can overcome your strong expectation about something by studying something, debugging a program or whatever.

> I think most scientific information that comes to us through the media is reliable

Your general point is well taken but is severely and gravely undermined by the belief expressed in above.

Yes. Even if the original information is correct it tends to be quite distorted by the media (the headlines are almost always hyperbolic), let alone when it gets picked up by the public and repeated.

It depends on how you define the “scientific community.” If your kid has a staph infection, there are people who make antibiotic creams that will make it disappear in days. They’re scientists. Their science gives them power you don’t have to “believe in,” because you can see the results.

Most people with a Ph.D. in a field with “science” in the name aren’t scientists. They work in fields that don’t have the same level of rigor as nuclear physics. (My degree is in aerospace engineering, and many real scientists would consider us bumpkins in how comparatively undisciplined our field is compared to their’s.) Those fields confer little to no power to produce tangible and undeniable results.

Like how North Korea is a ‘democratic republic.’ If you have to put ‘science’ or ‘evidence based’ after your discipline, it’s probably not science (or science with lots of problems).

See: Social science Political science Evidence based naturopathy/chiro

> Most people with a Ph.D. in a field with “science” in the name aren’t scientists

Computer Science. ;-)

Perhaps an appropriate assessment, given that CS seems to be half engineering/tinkering, half applied mathematics, and half nonsense.

Your last point is critical. People at the end of the 20th century had come to “trust the science” because it conferred tangible power on those who wielded it. “Science” could send a man to the moon or a bounce a phone call off a satellite to the other side of the world. Your average person doesn’t have to understand the rocket equation, or trust NASA. They can watch a launch in Florida and see with their own eyes the awesome power of “science.”

Then, folks started invoking the authority of “science” in connection with disciplines that don’t confer tangible power. For example, if “education science” worked, we would know it. It would convey power the results of which people could see with their own eyes without needed to pore through studies, or putting any faith in “education experts.”

You're right about this. And it makes discussion, much less "debate", so difficult. I'm pro-5G but willing to entertain the anti-5G evidence because what if there was some and it was true or at least unexplained?

One problem is that often science says nothing about policy.

_Science_ doesn’t say we should ban fossil fuels, but it does make predictions about what will happen if we continue to use them at the current rate.

It’s not science denial to be against the policy.

yes certainly science can inform policy decisions. How do I put this...

Science has nothing to say about what we should set as the objectives of our policy. Science can, however, inform our approach to attaining that objective.

To oversimplify my view:

Politics = applied philosophy Science = applied epistemology

Pretending that science can create policy leads to things like eugenics (but Darwin said it would make us fitter so it must be ‘good’!)

There's another important part of this though, oil companies pay scientific researchers to publish views against the veracity of climate change, research and predictions. That's why they say it's science denialism because those people are making shoddy arguments. Disagreeing about whether you care about global climate change is not science denialism but making up things to say there's no science is denialism.

That’s an important distinction, but is lost on 99% of the population.

Try to voice any skepticism about climate change policy and most people will call you a climate denier.

Same with vaccines; express a skepticism of efficacy and suddenly you are an anti-science anti-Vaxer.

The irony is that skepticism and asking annoying questions is essential to science. Ignoring ‘anti-scientific’ arguments is inherently anti-science.

Maybe assess the argument by the first two studies. If either of them do not support the point (or as has frequently happened for me, actually oppose the point) then you can reply for posterity and no one else needs to replicate your debunking effort.

“Your first two studies not only don’t support your argument but invalidate it, and I stopped reading after that.” ends most disagreements.

You will have probably done more work than the poster who often has just discovered keyword searches and related articles in google scholar or science direct, but everyone will learn something.

What are some examples of the claims people made vs what the research actually demonstrated?

mRNA vaccines are completely safe - Obviously false just like it is false for any vaccine since there are always going to be rare side effects in some members of the population. They very well may be just as safe as any other vaccine, but the truth is there are some things we won't know until a lot more time goes by. You can't say what the long term side effects are of something until it has been around for a long time.

Wind power is better for the environment - Similar to above, we don't really have data on the full lifecycle environmental costs of large scale wind farms. This is especially true because a good portion of the impact depends on how long the turbines last, what type of repair/recycle technologies are developed, and what type of policies get applied to recycling them. We see some things that look promising but others that are concerning.

Saying "mRNA vaccines are completely safe" is obviously not pedantically true. The same could be said of anything, not just mRNA vaccines. Nothing is ever absolutely 100 % safe.

It would, at least in theory, be better to realistically say what we knew and didn't know about the risks.

But public communication about those kinds of risks and tradeoffs is hard. If you say there might be any kind of a risk at all, some people are going to get hung up on that or grossly overestimate those risks compared to the benefits (or to the risks of not getting the vaccine -- also not something everyone would be affected by but clearly a non-zero risk).

Taking that into account, "completely safe" might be a better approximation than most other ones you could make.

While I understand what you are saying, I would disagree. Saying that something is completely safe when it isn't (as you say nothing is completely safe) makes it appear that people are being lied to when someone does have some type of reaction.

I think we are better of thinking of risk in terms of every day risk management. Comparing risks to things like driving 10 miles, getting hit by a meteor while sleeping, etc. can help create a better understanding of the risks.

I'm not sure I agree with it either. I'd personally much rather take a realistic estimate (in cases we have one anyway) than a simplified half-truth. But I think I can see a rationale for why some people doing the communicating may opt for the latter.

Comparing to something else that people might have a more realistic intuition of sounds like a good idea.

I'll often read through the studies in an argument but it is very unfeasible to keep up with these people because they can just keep throwing crap at you, and if you disagree with the study, they'll throw another at you until you give up. I remember reading through one 80 page paper on how a public option for healthcare would bankrupt us, but if you actually read through the thing, it never accounts for all the money saved by not being spent on private healthcare (which would fund the entire thing). My local power company had a similar situation where they posted a huge study explaining why solar was costing them a bunch of money, but if you actually read the study it explains that it considers lost income from solar as costing them money (in the same way that moving to a more efficient AC system would "cost" the power company money). It's all bullshit, and it's too easy to abuse studies to defend your position.

It is sort of like how when some people want to prove something is true they quote the Bible. I find the behavior especially strange when they are quoting the bible to prove the bible is true.

If someone replies within a few minutes with a link to a study, you can usually just assume it doesn't even support their point unless they're actually a researcher in the field.

It takes way longer even for experts to read and digest a paper thoroughly to evaluate the methodology and results.

Googling for 5 minutes and skimming the abstract for keywords that might be related to whatever you were arguing about is usually the norm online.

It takes years of training and feedback to learn to properly evaluate a paper in your own field, and that's after years of undergraduate education at least. Most people online skip even the basic textbook level background and think they understand what they're reading.

It is not better in natural sciences. Alzheimer’s disease treatment based on the amyloid plaque hypothesis being the prime example. Basically researchers closed ranks and doled out grant money disproportionately to the amyloid hypothesis researchers.

This is sort of like adding momentum to a gradient descent algorithm, though. It is rational congeal support around a plausible hypothesis to see if it pans out rather than pursue all hypotheses (whatever that would even mean) in a desultory fashion. In the case of amyloid plaques there may have been some academic misconduct involved as well.

Sometimes I feel like non-scientists have this idea that scientists should be super-rational actors and never be bamboozled or be wrong, but frankly, that isn't realistic. Scientists are fallible, have finite time, and are subject to social trends and pressures like the rest of us. Expecting that they always get it right is unrealistic and not good for science.

> It is rational congeal support around a plausible hypothesis to see if it pans out rather than pursue all hypotheses

Is it though?

Yes. At least there is a plausible argument for it in many contexts. Imagine a suite of ten hypothesis, one of which is correct but each of which requires 6 effort units to prove or disprove or discover or whatever. If we only have a 10 units of effort to devote to research, we won't get anywhere if we divide them equally but we can eliminate each hypothesis if we work serially.

In reality I would expect the effect to be even more pronounced because progress on a hypothesis is almost certainly non-linearly related to the number of people working on it (with some point of diminishing return, of course). It is totally reasonable for a community to more or less work on proving or eliminating the few most reasonable hypotheses at a time rather than to spread themselves over the (frankly enormous) space of hypotheses relevant to a particular area.

I'd hardly argue that our system of allocating scientific research effort is perfect, but how could it be?

> Imagine a suite of ten hypothesis, one of which is correct but each of which requires 6 effort units to prove or disprove or discover or whatever. If we only have a 10 units of effort to devote to research, we won't get anywhere if we divide them equally but we can eliminate each hypothesis if we work serially.

A model of research without evidence demonstrating its accuracy is not convincing.

You might claim that we could never have discovered the Higgs without such coordinated efforts, and I could counter that without such coordination, we might have sooner discovered and advanced alternative paths that require considerably less funding and coordination, like wakefield accelerators.

I don't think science that advances by stepwise consensus will outperform random, independent search in general, but only in very limited scenarios.

> It is totally reasonable for a community to more or less work on proving or eliminating the few most reasonable hypotheses

Only if "the community" consists of largely independent thinkers that reach their own conclusions, and are not influenced by fads or celebrity personalities. That's the only way to actually ascertain "the most reasonable hypotheses", and unfortunately, scientists are not immune to such influences.

I think current funding mechanisms for science select for non-independent thinkers. At the beginning of the scientific era, most researchers were funded by patrons, which supported much more independent research (at least, independent of other researcher's opinions, not the patron).

Scientists should not shut out other ideas for two decades even though there were serious challenges to the hypothesis the whole time. Part of being a good scientist is admitting your chosen hypothesis isn’t correct and that it’s time to create a new theory based on the experimental evidence available. Otherwise, it’s just a research based cult.

This almost never happens, to my knowledge. Most fields have people pursuing alternative hypotheses even when one thing is in vogue. For example, in the case of Alzheimer's Disease there were and are a lot of other threads of research and I would be that your typical AD scientist would have had a good grasp of many of them regardless of the research they were focusing on.

https://biology.stackexchange.com/questions/3223/what-altern...

People relying on a salary are rarely free to pursuit the truth

My website ran into this issue. I came to a conclusion that was like 80% correct, but it was 20% missing. Heck, I think I solved that 20%, and I found an additional 20% I feel is missing.

Anyway, there are a few reasons to continue solving. First, its good to grow knowledge and make correct decisions. Second, someone smart will call you out on it at some point, you will not be able to escape it.

I am casual, so I have no reason not to keep pursuing it, no PhD or grants needed. Heck, my audience loves when I dive deeper than before. My website produces science and I have quite a few academics reading it. Wonder why this doesnt apply in Academia?

Sociology lives in an uncomfortable place at the intersection of "important" and "difficult". Forming a rigid experiment on human beings is vastly more difficult than a simple thing like an atom or a mineral, and so much worse with groups of human beings.

STEM people like to dismiss it because it doesn't produce the same kind of rigid results and is therefore useless. But sociology is nonetheless important. Like it or not, we have to make decisions about how the world will run, and the fact that we don't have perfect information doesn't let us opt out of that.

Combine that with all of the usual human failings -- pettiness, meanness, closed-mindedness, greed, etc -- it sounds impossible. But it does, slowly, gather data and formulate theories.

This isn't made better by the fact that most STEM people imagine they can read primary source material and understand it -- a mistake they wouldn't make for a "hard science" in a different discipline. Like every field, the frontiers are based on huge amounts of background material, which is even more vast for a field that's more complicated than the nice, neat laws of physics or chemistry.

None of that excuses those human failings of sociologists. They need to do better, and hold each other to account (and not just to foment their own failings in their place). But we do need to recognize that this work is important. The world is complex and difficult and we'll make better choices if we try to understand, rather than dismiss it as unknowable.

While I agree that there is some work in the field that is important, there is an absolute deluge of ideologically driven garbage. There is also a lot of garbage of the "yeah, duh, of-fucking-course" variety. studies like "negative interactions with community reduce feelings of belonging" ... uh yeah, no shit sherlock. And then even the more important stuff cannot be investigated in the same rigorous manner as other scientific disciplines. I think we just need to stop calling sociology papers "scientific". Fundamentally, they are not, and should almost always be taken with a massive grain of salt, and they damn sure shouldn't be influencing public policy decisions to the degree that they currently do.

> There is also a lot of garbage of the "yeah, duh, of-fucking-course" variety. studies like "negative interactions with community reduce feelings of belonging" ... uh yeah, no shit sherlock.

I think these studies are useful, first of all to test whether it's actually true because sometimes "well du'h" turns out to be wrong, but also to quantify the exact effects. Is it a large effect or a small effect? How large exactly? Which factors exactly contribute to this effect? What exactly is the breakdown of the effects? It might be possible that 20% of the people are effected by it and 80% of people are not; or perhaps everyone is effected by it.

There's often all sorts of non-obvious nuance that's possible, which can be very significant.

Yes, that is true. However, I would posit that the actual result of the most well researched, scientifically backed, rigorous results in all of sociology essentially amounts to empowering governments and private companies to produce more effective advertising, propaganda, etc.

That seems overly cynical. At the end of the day sociology is like any other science: "find out more about the world". Doing that is rarely a bad thing.

> There is also a lot of garbage of the "yeah, duh, of-fucking-course" variety. studies like "negative interactions with community reduce feelings of belonging" ... uh yeah, no shit sherlock.

The importance of these studies are often not the expected results, but the average magnitude of the effect, and the parsing out of confounders.

Do chronic minor negative interactions have more of an effect on feelings of belonging than a single major negative interaction? This has policy implications.

What are the effects of negative interactions on highly social versus highly non-social individuals? What sorts of coping mechanisms do these two very disparate groups use to deal with negative community interactions?

I can think of a bunch more questions answerable by this type of research that don't have obvious answers. The importance of these questions depend on the magnitude and confounders of the original question.

> I think we just need to stop calling sociology papers "scientific". Fundamentally, they are not

They are, or are capable of being, as scientific as Darwin's crude observations of finch phenotypes.

I also do not regard Darwin as particularly scientific, though he broke open some flood gates for very scientific research.

And your comments on policy implications are precisely what I am saying we should avoid - Why would we set policy based on studies which 1) are not scientifically rigorous (based on self reporting, surveys, small population, low ability to control confounding variables, etc) 2) do not actually suggest that policy would be effective in rectifying the problems identified in the study and 3) do not necessarily identify problems (e.g. is it really the business of the government to set policy with the aim of optimizing some self reported individual metric, such as feelings of belonging?)

> Why would we set policy based on studies which

We shouldn't. If self-reports are ever used to set policy (and they should be) this should be on an individualized, ad hoc basis.

If a squeaky wheel comes to you, oil it. Maybe ask around if there are other squeaky wheels that no one in power is paying attention to. Oil them too. But don't go around oiling every wheel as a matter of policy, as you will end up with a bunch of overly oiled wheels having problems from over oiling.

Plenty of sociological studies (such as education interventions) aren't based on self-reports, but tested results.

> is it really the business of the government to set policy with the aim of optimizing some self reported individual metric, such as feelings of belonging?

The general governmental purpose here wouldn't be to make everyone feel like they belong, but to decrease as much as possible mass shooters, abusers, and the like. And to make it easy for people to report problems they are having, or for outsiders to discover problems. The Turpin case could have been nipped in the bud if the adults and children who noticed how unkempt and smelly the oldest daughter was when she was briefly publicly schooled had intervened (https://en.wikipedia.org/wiki/Turpin_case ).

The problem I have with your take is you mistake the voting population as a bunch of libertarians, something that is a very commonly held viewpoint here on HN. They are not.

Voting blocks have what they see as problems they want to change. If you come at them with sufficient evidence for a plan that may work to change the problem, there is a good bet they'll vote for it. If you decide science is too hard and that we shouldn't do that silly science stuff, they will line up right behind the next authoritarian that says 'make america simple again' and enact devastating plans that they believe will solve the problem.

I mean, you can get in front of the voting block and tell them that status quo is just fine if you like, but expect it to be hard work and don't expect much success.

I think I've learned the most about life as a citizen from the sociology lessons and a book written in late 70's or 80's. The stuff they've concluded then still stands today and I'm sure will still stand as long as there are humans. Such and eye-opening discipline and I remember it with fondness. Always surprised when people start bashing sociology for reasons unknown to me.

> It is also well known that papers that support certain "sides" tend to be easier to publish (and in higher ranked journals), plus can yield more media attention.

Covid-19: Lancet retracts paper that halted hydroxychloroquine trials

https://www.theguardian.com/world/2020/jun/04/covid-19-lance...

I'm not sure if that particular study is a good example of this as everything and its aunt was getting published about COVID back then. It is disappointing that other studies were shut down on the assumption that the mortality results from such a limited (and fatally flawed) analysis were correct, especially given previous knowledge of the side effects and dosing considerations of hydroxychloroquine.

“ thought of research less as a way to determine the truth, but just as a method to influence policy and public opinion. If we thought something was 80% likely to be true, there was pressure to "close ranks" and pretend as though it was 100% true, and to avoid publishing anything that contradicted it. ”

That seems to be the general spirit of this time. People feel the need to take sides and then make sure their side wins. Same happens in journalism. Most journalism these days seems to be about supporting a viewpoint and less about conveying neutral or complete information.

Sad that scientists also feel that they should be activists.

If you are a physical anthropologist there are many questions you can not ask, and many findings that people do not want to hear. Tread carefully, because one wrong step can end your career.