Debunking Cloudflare’s recent performance tests

Re:Debunking Cloudflare’s recent performance tests by deadmutex on Tuesday December 07, 2021 @01:33AM (#29469614)

Unless your competitor was offering a complimentary service to do that for all customers (tuning) then perhaps it shouldn't be done. I believe the idea of benchmarking is to get a sense of how the service would perform for a customer.

Consider the following scenario: You calling your ISP that you're going to do a benchmark. Then the ISP gives you the best service, at the expense of other customers bandwidth, for the duration of the test. Then after the test, the ISP removes the QoS modification, and reverts to the old behavior. That could end up with a biased, unrealistic result.

[ Parent | on HN ]

Re:Debunking Cloudflare’s recent performance tests by dekhn on Tuesday December 07, 2021 @10:21AM (#29473348)

No, benchmarking aims to identify the peak performance of a service, not the average customer experience. Real Benchmarking is done for people who are serious about performance. marketing benchmarking is designed to attract naive customers.

As for the latter, that benchmark wouldn't reproduce and would require extensive configuration (which in a sense is just a part of negotiation between the customer and the client). If a company got caught faking benchmarks run by competitors and it could be shown publicly they did, that's pretty much the end of that company's public reputation.

[ Parent | on HN ]