Except for the ones that conclude it is all a scam.

There is genuine merit in peer review though. A lot of the marketing "papers" from OpenAI and Google would benefit from going through a peer review process instead of scholarwashing and uploading straight to Arxiv.

One simple example, (from memory) the Bard paper doesn't include results for experiments in which GPT-4 outperforms it. As a result, people come away from these works with an inflated understanding of their capabilities. This wouldn't pass peer review.

Industry doesn't need peer review. The proof of the pudding is in the sales.

This is the final stage of capitalism, where market success is conflated with scientific rigor.

Engineering, not scientific.

High Sales = A large number of people can attest to the engineered good or service being of high enough quality that they will exchange hard earned money for the ability to use it.

Peer Review = Some folks who derive self-worth from citations ask you to add a citation to their work and you do it because you'll probably need to ask them for the same some time later.

This doesn't really account for fraud or abuse. Plenty of profitable companies whose product is a scam, or has other factors that make it successful.

Market success isn't the same thing as validity.

Peer review isn't the same thing as validity either. Both are proxies, sales are simply a much better proxy. Especially long term sales from an established company.

Look at it like this: a Peer Reviewed (tm) article comes out saying "Foos cannot Bar, it is impossible". The same day, Apple releases "Bar for Foos, $8/month". Over the next year you see media outlets discussing how well Foo Barring works. Online reviewers talk about how they've incorporated Barring their Foos into everyday life and it has benefitted them in all these great ways. Your colleagues at work mention their fruitful Foo Barring adventure over the past weekend. Routine posts on HN come up where hackers describe how they've incorporated the Foo Barring API's into their own products in some novel way.

Your mother then calls you up to ask if she should get involved with this new Foo Barring thing. What do you say, can Foos Bar?

A quick trip around audiophile companies should quickly disabuse you of the notion that high sales implies any kind of worth to a product. There are several companies making lots of money on selling gold HDMI, ethernet etc cables "to improve sound quality", and plenty of rubes buying them and "hearing the difference".

Real Hi-Fi companies like Sennheiser and Bose make far more money than people selling gold HDMI cables. And they've been making much more money for decades and will continue to make money for decades.

Grifters don't win in the long-term.

>Grifters don't win in the long-term.

Plenty of religions suggest otherwise.

(Dear reader: I'm not referring to your religion - I'm referring to the other religions.)

Wow so edgy and cool. Are you a teenager?

You frequently see this kind of reasoning in medical quackery:

> Butt-candling[1] must work, just look all these happy customers!

But history is replete with ineffective or downright harmful treatments being popular long after the evidence showed them to be ineffective or harmful. Homeopathy is a prime example of this, seeing as those concoctions contain either no active ingredients or (in cases of low dilutions still labeled "homeopathic") contain ingredients picked based on notions of sympathetic magic ("like cures like").

[1] A hopefully fictional example.

This is a perfect example. In the real world, some placebos work. You take them, believe that they will work, and they do. Fantastic! Your problem has been solved, you share your story with friend(s), the cure spreads based on how well it works for them.

It's only in academia that they must instead work in a sterile environment that some person who has never stepped foot off a school campus thinks is "more authentic" for them to be seen as legitimate.

Then why does it dress its marketing up in academic paper format?

In want of sales.

It gives a nice patina and some legitimacy. They have no need to publish quasi-scientific articles.

You can often see a p-hacked study from a mile away because they measure large numbers of unnecessary variables. They then pick those 4 variables that yield a probably false signal and publish on it. One would think these would never pass peer review, but they're regularly peer reviewed, published, and then, shockingly enough, fail to replicate. Hypothesizing after the results are known falls in the same bucket here. This is why pre-publishing kicked off, yet it's also hardly a savior.

The point I make is that peer review can not be guaranteed to 'fix' science in any way we might like. The Sokal Affair [1] has now been replicated repeatedly, including in peer reviewed journals. The most recent one even got quite cheeky and published it under the names "Sage Owens, Kal Avers-Lynde III" - Sokal III. [2] It always preys on the same weakness - bias confirmation.

[1] - https://en.wikipedia.org/wiki/Sokal_affair

[2] - https://www.nationalreview.com/news/academic-journal-publish...

What is honestly a lot more compelling than peer review is multiple supporting pieces of evidence from different sources. One result might be spurious, but if you can find a couple independent studies showing as much its probably a real phenomenon.