But the same point applies when comparing against Lambda. It's not a cold start if the app code is already resident in memory in advance. Workers and Lambda also proactively load some apps in advance, and we don't call it a "cold start" in those cases.
> But I agree everyone should run their own benchmarks.
But they literally can't because the ToS forbids it.
Benchmarking accurately is hard, so CF trying to make sure they're not going to be subject to a "report" by someone whose tests are garbage, or by a competitor who's benchmarking poorly to make their own offering look better isn't totally unreasonable. Ideally, as far as CF are concerned, they'd be so far ahead of the competition it wouldn't matter, but presumably that wasn't the case when the ToS were written. The fact the CEO posted to say the clause is being removed means we can all freely test in future.
This is how things are supposed to work. Someone does something, someone else reacts, and the first person updates their position. Expecting everyone to get things perfectly correct on the first try is nonsense.
I dont think Fastly has ever explicitly called out a 35us against Cloudflare 5ms. ( Please Correct me if I am wrong ) They may have some marketing material about 35us being faster than the rest of industry. But that doesn't imply it is Cloudflare because there are lots of player in the industry. First name comes to mind would be AWS.
Compared to Cloudflare blog post directly naming Fastly in a benchmark is completely different.
Second being Fastly was unhappy about this test and posted on twitter for weeks even quoting and mentioning @cloudflare.
But I agree everyone should run their own benchmarks.