Re:Debunking Cloudflare’s recent performance tests
by shah0048 on Friday December 10, 2021 @02:19PM
(#29514254)
No it isn't. It's their response to CF's crap test which CF then posted on their blog <---THAT is mud slinging.
And Kenton himself agrees: "On a serious note, I've been saying for decades that benchmarks are almost always meaningless..."
Benchmarks can be used to mislead which CF was clearly doing and which Fastly pointed out with their "Lies, damned lies..."
Re:Debunking Cloudflare’s recent performance tests
by kentonv on Friday December 10, 2021 @06:14PM
(#29516629)
I think you misinterpreted me a bit. I don't believe CF's benchmark was designed to be misleading or biased. I think the benchmark had a perfectly reasonable setup and was actually representative of one relevant use case -- the case of an app that just returns a response without a lot of processing. I think Fastly's criticisms of the benchmark are surprisingly weak compared to the strong language they are couched in, and the strong language itself is what I refer to as "mud-slinging".
My point about benchmarks in general was that unless your application is extremely similar to the one tested -- which is unlikely -- then these numbers don't really tell you much about how your own app will perform. This is a criticism I have of every benchmark that attempts to compare performance between very different technologies. I don't think it makes such benchmarks unethical, just not very useful.
The post's current title is Lies, damned lies... and the article's synopsis is,
> This nonsensical conclusion provides a great example of how statistics can be used to mislead. Read on for an analysis of Cloudflare’s testing methodology, and the results of a more scientific and useful comparison.
That is definitely mud-slinging.