Here is the thing, at least in the case of database software, the people that are competent to set up a benchmark correctly are fairly rare. The people that have the budget for the hardware are even more rare. The last thing the world needs is for the web to be polluted by so many incompetent "benchmarks" performed by the clueless that you can't find the authoritative information that you need to do your job.

Take a wild guess about how much memory and how many disk spindles it takes to get a TPC benchmark on a modern server to be CPU limited. Take a guess as to how many weeks of engineering time it takes to perform and validate the benchmark after getting a pile of hardware delivered to your lab. The average blogger writing 3 posts a week ain't gonna do it.

There is always a well sounding rationale attached to censorship.

It also prevents meaningful benchmarks to be published.

And it's a self-fulfilling prophecy if we don't allow inexperienced benchmarkers to learn how to run meaningful tests.

I'm on the fence about the benchmarks but I think this is a generic broad brush to sweep away the parent's comment given the real concerns they've presented and you've chose to not address.

Do you really thing any agreement that involves silence is censorship NDAs included?

Sure, it's easy to simply say: sucks to be you! Especially to "community bullies" like Oracle. Even just between semi-competing open source projects benchmarks can be so bad they border on libelous.

Oracle has just enough budget to hire someone knowledgeable to engage in the discussion, pointing out the issues and their fixes, possibly even linking the benchmarks that were (in their view) properly performed. One good engineer. Compare with the legal fees required to draft such an appalling, scared and knowingly guilty clause as "you can't talk about how our product performs despite having paid for it." This really needs to be treated with total derision, and utter contempt. It's an approach that has earned that response for it's sheer laziness before even beginning to measure it against any ethical standards.

Oracle are just an awful company in every dimension. This, is just like most everything else they do in that it reinforces the reputation they earned for themselves.

Except they started the practice in response to an academic researcher publishing, while simultaneously trying to get the researcher fired.

https://danluu.com/anon-benchmark/

But this doesn't mean having the vendor control publication is going to get you any benchmarks that are better. We can predict pretty easily that the benchmarks they publish will be skewed in their favour.

On the other hand, we can't predict what solutions society comes up with for dealing with incompetent benchmarks. For example, you say professional bloggers would do lousy jobs, but this is exaclty the kind of things that neckbeard hobbyist bloggers would take upon themselves and do excellent jobs at.

That or other solutions could emerge, but the whole point of censorship is to prevent that.

Well, most times I'd like to know how a badly configured X compares with badly configured Y, on <random hardware> - because that's usually how things are run.

> Here is the thing, at least in the case of database software, the people that are competent to set up a benchmark correctly are fairly rare. The people that have the budget for the hardware are even more rare. The last thing the world needs is for the web to be polluted by so many incompetent "benchmarks" performed by the clueless that you can't find the authoritative information that you need to do your job.

If these apparently incompetent people running the benchmarks are also the incompetent people creating the schema's and queries out in the wild then the benchmarks are still worthwhile.

I want to know what performance I can expect, not what I can expect when the environment is highly tuned by rare experts that will never touch any of the systems I have to deal with.

I believe Microsoft does the same with SQL Server.

https://www.reddit.com/r/programming/comments/2mhpwp/postgre...

Ta for the link. That is quite a discussion. The first thread is priceless (it turns out that grauenwolf isn't quite as authoritative as they first appear - fancy!)

Never underestimate how keen people can become about something they have paid £1,000s for. How on earth could a DBMS like MySQL/MariaDB or PostgreSQL possibly compare to the mighty Oracle money generator or MSSQL ... errrr ... money generator.

Just in case you wonder whether open source DBMSs should be taken seriously, try reading this: https://postgresweekly.com/ and this: https://mariadb.com/resources/blog

I worked for a competitor and we had similar verbiage. It's there to prevent "XXX is a 10 million times faster than YYY." We actually gave everyone permission if they both asked and allowed us to review their methodology to make sure they weren't doing anything horrifically wrong (benchmarking is hard).