To clarify, Fastly did not perform any benchmarks on Cloudflare's network. We ran our own benchmark, comparable to the one that Cloudflare ran on our platform, and compared those results to the ones that Cloudflare published as part of the original blog post.

The Cloudflare terms prohibit benchmarking without their explicit permission.

> To clarify, Fastly did not perform any benchmarks on Cloudflare's network. We ran our own benchmark…

Am I the only having cognitive dissonance here? I was confused how they complained about not being able to perform benchmarks but then… ran a benchmark? There must be some distinction here that I’m missing

Fastly is unable to reproduce the benchmark against Cloudflare. They reproduced a modified (and theoretically improved) benchmark against their own service, but then had to compare it to the original values provided by Cloudflare.

CloudFlare ran a benchmark on both services and published their results. The contention is that the benchmark did not represent fastly performance fairly, so fastly reran the benchmark in what they consider to be a fair way, on their service only. They did not retest CloudFlare in their benchmark. The results in the post compare CloudFlare's results for CloudFlare and fastly's results for fastly

Yeah, ok that must be the distinction.

However, I see that the TOS says you may not "perform or publish" benchmarks. Seems this still violates the "or publish" part, even if Fastly avoided doing the performing. What would be the point of forbidding publishing if it is already forbidden to perform said benchmarks in the first place? The TOS seems to also forbid publishing on the benchmark even when someone else performs the Fastly part of the benchmark.

Edit: It must be the TOS doesn't apply if you don't use the service.

Fortunately, no one needed to agree to the CloudFlare ToS to read their blog post.

Haha. Actually, now that I think about it, I guess the TOS doesn't apply if you don't use the service!

Sorry, I was trying to clarify in my comment but failed to put it through clearly. When I say "We ran our own benchmark" I mean Fastly ran a benchmark on a Fastly Compute@Edge service.

Notably, the self-service and enterprise TOS differ a bit here.

From "Cloudflare Self-Serve Subscription Agreement" https://www.cloudflare.com/terms/ , "2.2 Restrictions. Unless otherwise expressly permitted in writing by Cloudflare, you will not and you have no right to: [...] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;"

From "ENTERPRISE SUBSCRIPTION TERMS OF SERVICE" https://www.cloudflare.com/enterpriseterms/ , "2.3. Restrictions and Acceptable Use. Customer must not: [...] (i) perform or publish any performance or benchmark tests or analyses relating to the Service, other than solely for Customer’s internal use;"

Generally, it sounds like the kind of thing that would be obvious to a lawyer (we want to prohibit random people badmouth our product, right?) but stupid to engineers.

The best outcome would be if Fastly and Cloudflare both dropped any language limiting benchmarking and sharing of results.

If they want to protect their brands, the most common sense requirement would be "We require you to tell us, within X hours after publishing, of benchmarks you're publishing and send us a copy of the article in which you published them (or link if publicly accessible)."

They stated in the blogpost that they only performed the benchmarks against their own services

> So instead, we ran the same tests (echoing headers and measuring TTFB via Catchpoint) against our own platform