It's really hard to benchmark this sort of a thing. There are so many layers of caching and external factors that play into it, from all manner of sources including the operating system load and configuration, disk firmware, hardware configuration, and so forth; and the harder you try to isolate these effects, the farther you get from a realistic benchmark because all the factors that were removed are affecting real world performance in a big way.

This is a big reason why for a long time many large DBMS-providers had clauses in their licenses prohibiting 3rd party benchmarks. You can fairly easily construct a benchmark that makes any given DBMS seem great or awful, and there's no such thing as an objective test.

Fully agree with this idea. All tricks can be a real world strategy and it is impossible for anyone to claiming that they have an absolute fair benchmark.

So the only way we can do to approaching it is to provide more real-world-like cases and forget all tricks vendors might play inside their systems.

Also, people will concern the representative of the cases benchmarks provide. So we plan to make this benchmark more like a framework to support customized cases in the next step.

Previous discussions from last time they posted this: https://news.ycombinator.com/item?id=36856815

If they’re going to rank themselves so much higher than their competition, they might as well call that out up front and explain why the discrepancy is so large.

Yes, of course vendors have bias. But IMHO, if a benchmark is reproducible and the use cases can match users' needs, then we can say it can somehow help decision making.