This is the main problem with Compute@Edge, "for years".
They have been advertising publicly C@E for years but it was a restricted private beta. In my opinion the state of C@E has historically been falsely advertised. You could not go to the website and sign up, put in your credit card details and use C@E. If you contacted support they told you it's not generally available, it's for exiting customers and no more seats are left, it was at capacity.
After I bit of Twitter ranting one of the C@E advocates at Fastly did reach out and did get me test access and he did genuinely help as much as he could. I did like what I saw and in my personal testing found Fastly faster than Cloudflare Workers using Rust / Wasm / Wasi. Also one of my services where having issues in Cloudflare workers with ByteBuffers sending binary data, I don't know why but it just worked in Fastly Wasi runtime.
Regardless today my preference would still be to use Cloudflare. You go to the Cloudflare website, sign up with minimal information and are then good to go. Need to move outside of the test tier? put in your credit card, click a button, done. The pricing for Cloudflare is transparent and hugely attractive for workers. Fastly, there is no clear pricing, they are applying there CDN pricing model with a $50 minimum fee which you need to go and read in the small print. That is ok for enterprise clients but obviously they are missing up on all the pay as you go small scale devs that Cloudflare attracts and all the other pay for what you use PaaS's attract that then take it in to their day jobs becoming advocates where the big contracts are signed.
Fastly need to decide if they are a CDN or a internet company with a CDN product. They seem undecided what they are at the moment and have poor product structuring and it's genuinely very hard to go put in your credit card and give them money! Cloudflare are quite clear they are a internet technology company now not a CDN and make it extremely easy for people to give them money in a few clicks.
I greatly respect Fastly and their technology, but they (and Varnish) are biased against small companies and make it impossible for me to sign up customers who would benefit from their services and one day might grow into big spending customers.
AWS has shown that the days of "min $5000/year" (as both these services have quoted me for individual products) can be over and you can be profitable. Companies who don't catch up with that are cutting off their future.
I wish someone would smack Okta over the head with this and get them to figure that part out.
Yes, they're expensive, but the per-use pricing was acceptable for what we wanted. I didn't realise there were (non-public) minimum spend amounts until we hit the end of our trial period and couldn't find any way to pay them other than to contact sales.
Sales wouldn't take the order for just a handful of users.
If you read the discussions in the financial forums Fastly might be in a tricky place since their biggest customer is assumed to be Amazon and if one goes to their website https://www.fastly.com/partners/featured/ they show off all the biggest cloud providers and customers. Seems like Fastly are afraid to steal customers from their partners.
Cloudflare's pricing is sneaky. They are good for small and personal projects with their free tier, but as soon as you need anything non trivial you get hit with their $xxxx/mo minimum commitment, "contact us for pricing" Enterprise plan. On Fastly you pay $50/mo (peanuts for a business) and that's it for basically anything you might want from a CDN, VCL is endlessly flexible and free. That $50 buys you access to a featureset that is arguably beyond Cloudflare's Enterprise plan. You will need quite a lot of traffic to go over $50, so overall it works out significantly cheaper for an SME that wants to use anything but a very basic CDN.
It's also important to remember that a lot of cases are perfectly served by VCL, I believe most businesses won't even need C@E. Sure, most people would need to learn a bit about it, but it's amazing to have it on your disposal for free compared to a usual rigid configuration.
Besides, something they don't advertise but is now the main selling point for me is support. Fastly was the only company I had pleasure dealing with where after shooting a technical question to their free tier support I got not just a technical answer, but a piece of code that solved the problem I had. In an hour. With a link to their Fastly Fiddle demonstrating how it works. When I encountered what looked like a bug, a few hours later their support person filed a PR to their TF provider fixing the docs. It was and is a breath of fresh air compared to my experience with any other SaaS, Cloudflare included.
I think the only thing that could've swayed me is Cloudflare's Chinese network. It's gated behind their Enterprise plan and they won't hold your hands much, but if you need it it's there.
This is a MAJOR problem in our industry. Products that exist, but don't actually exist (this sitiuation). "Platforms" that are only platforms if you've got a secret deal nobody tells you that you need to have (Twitter, Facebook). "Open" systems that close themselves off once you hit a very, very low threshold of use, and then force you to do a lot of not-very-open things to regain access... I could go on and on.
(I'll note that this is separate from the problem of AI-based fraud detection run amok.)
Most SaaS are structured in an effectively "non competitive market" kind of way. Lots of information hiding (don't post prices), high costs of switching (vendor lockin), and so on. These type of things stifle competition and lead to a more pseudo Monopoly kind of state rather than a truly competitive market.
Anyway, transparent pricing should be required for all businesses IMO. The problem is particularly bad in healthcare, and big contributor into high costs.
Of course, it will take a few decades for laws to catch up
In their annual reports Fastly have previously said C@E was available. They regularly released PR pieces of new functionality on C@E.
The reality was it was in private beta and deep in active development. Their official annual reports and PR pieces used by investors for investment decisions where misleading insinuating a product was available that was not. Fastly annual reports where painting a misleading picture. What's worse is the honest answer, we are deep in development and now have several large existing customers onboard for private beta is nothing to be ashamed of putting out to your share holders. Fastly where probably trying to hide the amount of time it's taken them to roll C@E out since first announced early 2019.
I'm sure every piece of public information went through their legal department and vetted but at the same time every piece of mandatory training I've done at publicly traded companies have me believe I could suffer huge legal consequences with the SEC for doing similar.
Are you going to remove the ToS clause or issue a correction on the blog post?
If I didn't state upfront that I was the tech lead of Workers, someone would (rightly) call me out for astroturfing.
The clause says "Unless otherwise expressly permitted in writing by Cloudflare, you will not and you have no right to: [...] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;"[1]
IANAL, but this seems to very unambiguously prohibit benchmarking Cloudflare's services unless you have written permission. I know you don't want to get into an argument on HN, but could you like... bring it up to someone inside of CloudFlare who would be capable of changing it? You can point to this thread about how this clause is generating negative publicity.
[1] https://www.cloudflare.com/terms/ section 2.2
An unusual and excellent CEO stance. Now I like Cloudflare even more.
No JS required, just feed all web requests directly through them where they can see all first party cookies, encrypted contents, etc.
Not quite at the level of a public figure (Pres. Biden can't go around making flippant comments) but more than being just a private citizen or "I just work here". No amount of disclaimer can remove that, and for better or worse it's part and parcel with the job.
Without this in your T&Cs I could create the account for them in a couple of minutes. And avoid doing a screenshare to walk those who fail through the sign up process.
If you're an employee, yes. If you're a consultant, contractor, freelancer or similar then you are a third party doing as you do it on behalf of your client (the first party). This is for UK law, and the distinction of first/thrid party is important when it comes to tax (see IR35 for the mess created).
Fastly made it sound like contacting Cloudflare is "impossible", yet here you and one of your top devs are.
I've seen you reply about ToS issues before (specifically over caching of non-html assets): https://news.ycombinator.com/item?id=20791605
You verbally allowed it in that thread but have you considered officially adding that into the next revision of your ToS too?
As for "stand-up" with Fastly, I believe the whole situation brings only negative consequences on both parties. I always become wary towards any service that posts comparisons with its competitors (or simply with services of similar nature).
Good luck and wise decisions to you.
This - I mean even the conversation prior to this message - surely constitutes "expressly permitted in writing by Cloudflare"?
I don't want the following to come off as unnecessarily argumentative, but regarding the ToS, I'm not a lawyer either, but my "ability to read English" interpretation of the section on "perform or publish benchmarks..." certainly sounds like it is prohibiting folks from doing their own side-by-side comparisons. Which is, of course, nonsense, because any engineer worth their salt would do their own analysis, even if they didn't publish it.
Just sounds to me like the CloudFlare lawyers got a little too aggressive to the point of absurdity, but I still think it's fair to call out CloudFlare for this.
Even without the ToS language, if you're really going to stress test a service, it's probably a good idea to give them a heads up, lest you get marked a bad actor.
I'm assuming this isn't the only overly restrictive clause in the contract. Maybe it's an anomaly in an otherwise respectful ToS.
We aren’t owed a historical explanation, and yet we’ll likely receive one with what I presume will be a TOS update blog post in a few weeks.
I feel like this is that moment where someone lays on the car horn because they want to be sure the other driver understands that they’re a bad person, and should feel bad about themselves. It’s not about making you right by their actions, it’s about making sure they know the depth of your anger at them.
That has little value here. It’s socially valuable in interpersonal interactions, but it’s a tire fire when left uncurbed at Internet scale, and becomes vitriolic and harmful to discourse.
I may have misunderstood your specific intentions and desires from the CEO, and if so, I apologize; but I stand by my point in the general sense for all of us.
The ToS explicitly say that performing any benchmark analyses or test relating to the services is not allowed without written permission from CloudFlare.
99% of the people reading and attempting to abide by the CloudFlare ToS are not lawyers, but as a rule contracts mean what they say when they say it clearly and unambiguously as this seems to.
The way I usually see people handle this, and what I do myself, is to both state your relation to the company and clarify whether you're speaking for the company. Ex: "I'm the tech lead for Cloudflare Workers, but speaking in my personal capacity..." or "I'm the tech lead for Cloudflare Workers (speaking only for myself) ..."
On top of that, Kenton is a frequent commenter here, and I've never gotten the "air of superiority" vibe from him where such verbosity would be necessary.
I think the nuance is that you are presenting yourself as someone with responsibility/authority/control over the subject of these benchmarks. As a comparison, consider wording that skirts taking up that mantle:
Full disclosure, I work for Cloudflare, but ...
Not trying to be argumentative and really don't have any hostile feelings or intent. I understand where you're coming from.. just providing my outside take on how the interaction appears. You're within your right to defend your product. Nobody seems to have a problem with that. But you also decided to throw mud on the pile, metaphorically. You admitted you aren't plugged into the back and forth, that's fine. But this isn't news for Fastly and it's hard to take your side in this discussion when your solution is "go do your own tests" which is exactly what Fastly is would like to do. They clearly call attention to your ToS preventing them from doing that in the piece we're discussing here.
He did not. Profiling how your particular app and use case runs on a given serverless provider is not benchmarking.
It's the legal department... And the CEO already mentioned that they are removing it and he gave a valid response/reason.
It seems that they will actually benchmark Fastly in detail now ( could be after another improvement week), which probably isn't what Fastly wanted.
Something definitely seems to be happening if you read their response and i'm awaiting it with actual stats!
https://news.ycombinator.com/item?id=29468771
@dwwoelfel that's what you wanted? :)
It's nice that eastdakota responded here, but he had two weeks since the original tweet thread from Fastly's VP of Eng calling out the problems with their benchmarking. They didn't respond or retract the blog post in those two weeks.
As a cloudflare shareholder (and a fastly shareholder), I want Cloudflare to act ethically and either retract the blog post or issue a correction.
You are insinuating bad will/faith and that's not the impression i observed.
It is unethical to leave that up after Fastly pointed out core issues with the benchmarking, like using a free tier that was rate-limited.
Incidentally, this means Fastly's blog post is currently displaying test results that compare the enterprise version of Compute@Edge against the free version of Workers. Granted, our bad for the ToS clause, but still.
Despite the strong language in their post, Fastly has not actually demonstrated that anything was intentionally biased or unfair in Cloudflare's test. They've only laid out their opinions as to what would make a more representative benchmark. That's a debate you can have about any benchmark, but that doesn't somehow make the original benchmark "unethical".
Cloudflare's free tier does not optimize for speed/performance, but for available ( = unused ) datacenter capacity based on location. Which makes it less fast than their paying tier.
Additionally, there will be an update soon as mentioned before, based on past comments.
There are other things that you ignore/are unaware of that are not even mentioned in Fastly's post. Eg. That cloudflare also optimizes their network for routing to denser cities instead of rural areas. That metric is not even mentioned by Fastly...
Note: i don't know any inner workings of them as I don't work there. It's based on what I remember from their blog about their SDN and performance weeks. I suppose it's applicable to this scenario, if i got the details right.
I dont think Fastly has ever explicitly called out a 35us against Cloudflare 5ms. ( Please Correct me if I am wrong ) They may have some marketing material about 35us being faster than the rest of industry. But that doesn't imply it is Cloudflare because there are lots of player in the industry. First name comes to mind would be AWS.
Compared to Cloudflare blog post directly naming Fastly in a benchmark is completely different.
Second being Fastly was unhappy about this test and posted on twitter for weeks even quoting and mentioning @cloudflare.
But I agree everyone should run their own benchmarks.
But the same point applies when comparing against Lambda. It's not a cold start if the app code is already resident in memory in advance. Workers and Lambda also proactively load some apps in advance, and we don't call it a "cold start" in those cases.
> But I agree everyone should run their own benchmarks.
But they literally can't because the ToS forbids it.
Benchmarking accurately is hard, so CF trying to make sure they're not going to be subject to a "report" by someone whose tests are garbage, or by a competitor who's benchmarking poorly to make their own offering look better isn't totally unreasonable. Ideally, as far as CF are concerned, they'd be so far ahead of the competition it wouldn't matter, but presumably that wasn't the case when the ToS were written. The fact the CEO posted to say the clause is being removed means we can all freely test in future.
This is how things are supposed to work. Someone does something, someone else reacts, and the first person updates their position. Expecting everyone to get things perfectly correct on the first try is nonsense.
I don't have the full TOS in front of me but wouldn't this violate it?
Then again, a Cloudflare employee just gave written consent in this post for us to perform benchmarks, so maybe we're all off the hook! :)
1.8 in https://aws.amazon.com/service-terms/
Here's some background from 2008 on benchmarking clauses and their precariousness: https://corporate.findlaw.com/business-operations/n-y-case-c...
Why do firms continue to make unenforceable claims? (My least favourite are restriction of trade clauses in employment contracts that are simply illegal, unless paid for, where I come from. But they appear all the time in contracts)
It's legal 'chicken'.
The EU e.g. has 93/13/EEC aka "Unfair contract terms directive" which bans a lot of shenanigans in ToS (e.g. mandatory location for law disputes). Or just writing in your ToS that "you agree that contract parties are not bound by the EU GDPR" for another example, like some companies do write, doesn't mean this is a enforceable clause. Another example, Germany recently made a law that mandates automatic subscription renewals - such as mobile phone contracts, gym memberships or online services - can be cancelled every month (there already was a law before that limited the initial subscription contract to a duration of max 2 years).
If there are some laws specifically that disallow general "benchmarking" I don't know, but I wouldn't be surprised if at least in the EU such a clause would be unenforcable. That sounds like a clear unfair one-sided advantage (the customer cannot check the services actually provided match what was promised). Publication of such results is another matter.
Also, I am curious about what you said about Illinois? Sounds bonkers. I mean I could imagine criminal prosecution for a set of defined, deemed specially bad violations that constitute crimes, like "hacking" and "sabotage", but not for things like e.g. "failure to pay membership dues in time" and other civil matters. Then again, I remember that case of some guy in Florida who couldn't keep his lawn nice and green, and didn't have the money to replace the lawn again and again, so the HOA took him to court, he was ordered to replace the lawn (which he still couldn't afford) and ended up in prison for a couple of days for "contempt of the court" until some friends and neighbors replaced the lawn.
Sorry, what? Where? How's that even possible?
[0] https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act
https://www.ilga.gov/legislation/ilcs/fulltext.asp?DocName=0...
What legal reasoning exists to give companies this kind of unchecked power?
>According to the court, DeJohn's claim that he did not read those terms was >irrelevant because, absent fraud (which was not alleged), "failure to read a >contract is not a get out of jail free card."
https://www.wilmerhale.com/en/insights/publications/click-wr...
At the very least, it is an indication of where they stand. They are free to take any customers they want, and if they say they don't want customers with a certain use case, that's good enough for me.
Or maybe its entrapment!
Same as when accepting a job, never trust someone who says a section of their contract/terms won't be enforced.
I might need to start looking for another provider...
I find this statement egregious: benchmarks are probably the most important thing in producing quality software and are often (outside of FAANG) the first thing to be ignored. I have the displeasure of working with extremely poor software for the majority of my career, where a simple benchmark and brain-power would illuminate the issues days-weeks-months before production issues occurred.
I also see "benchmarks are useless" used a lot by vendors who perform poorly in said benchmarks. Or use useless benchmarks (like TTFB which can easily be fudged) which is just underhanded.
I meant they aren't great for comparing two very different technologies. I've looked at a lot of serialization benchmarks, having once maintained Protocol Buffers and later Cap'n Proto, and it's amazing how wildly different the results are depending on the shape of the data you're serializing.
Many engineers, product advocates, and sales engineers/architects build careers, playbooks, and talk tracks around debunking or handwaving others (and sometimes their own) benchmarks.
The pressure to make a sale and your livelihood against a customer that is looking at products on a top ten scorecard is a lot of pressure
So, I would encourage everyone to run your own test and don't believe what either side posts.
---
Otherwise there would be no business for the Gartners and Accentures of the world if customers were actually savvy themselves.
Strange statement. Didn't Cloudflare start the mud slinging?
I have no shares in this. But Cloudflare publishing questionable benchmarks with competing services while prohibiting competitors publishing benchmarks with Cloudflare services is certainly not fair play. Reminds me of Oracle.
I imagine it's pretty normal for the front-facing blog/PR department to be a bit disconnected from development, but we can't entirely blame Fastly for "complaining" seeing as Cloudflare already uploaded a bit of a diss post.
Anyways I think I once read a phenomenon where corporate environments eventually reach a point where the outside world knows more about the company than the employees. Not sure if this is actually an example of that or not.
Also I agree these blog posts are all kind of meaningless and people should do their own investigation if they're considering alternatives.
For us/me XX works better because of...
As a dev, it sounds to me like Fastly's whole purpose is wrapped up in attacking Cloudflare whereas Cloudflare is focused on innovating.
This then makes your response look accusatory and helps further a broader narrative post about Cloudflare's behavior here, which it seems your goal is to dispel, given your statement on whether engineering was involved.*
A suggestion for something actionable that'll make things better in this difficult moment: email product counsel about starting the process to override your EULA and enable Compute@Edge to do benchmarking. It'll take forever and it's the only way to have an objective, evolving conversation about benchmarks, which your leads will (at least, should) want after this happened.
* intentionally vague here, in order to give you space to delete and leave no record
The post's current title is Lies, damned lies... and the article's synopsis is,
> This nonsensical conclusion provides a great example of how statistics can be used to mislead. Read on for an analysis of Cloudflare’s testing methodology, and the results of a more scientific and useful comparison.
That is definitely mud-slinging.
And Kenton himself agrees: "On a serious note, I've been saying for decades that benchmarks are almost always meaningless..."
Benchmarks can be used to mislead which CF was clearly doing and which Fastly pointed out with their "Lies, damned lies..."
My point about benchmarks in general was that unless your application is extremely similar to the one tested -- which is unlikely -- then these numbers don't really tell you much about how your own app will perform. This is a criticism I have of every benchmark that attempts to compare performance between very different technologies. I don't think it makes such benchmarks unethical, just not very useful.
But mostly importantly, kentonv's last paragraph.
At the end of the day, benchmarking for your use case is the only true north. Everything else, on all sides, is marketing fluff.
I'm not sure I see the point given how comment deleting works on HN.
You can only delete HN comments that have no responses, and that user's comment has a response tree of sufficient breadth and depth that it seems unlikely that everyone who has responded will delete their comments before the 2 hour edit/delete window passed.
The usual convention if you want to delete a comment but replies are stopping you from doing so seems to be (if the 2 hour window has not closed) to edit it to say "[deleted]" or something similar.
That still leaves your name on it, and so I don't see how having your name also mentioned in one of the replies is going to make anything worse.
Thanks for clarifying that!
To your point, maybe it is a problem.
Who knows. Life is messy, and pitching absolutes that you pretend are clear to everyone else in order to intensify criticism of a person for having any thought at all, actually only makes it harder to everyone else. (this is response to our friend here, not you)
I'm not sure why you're continuing to react strongly and personally to me. We shouldn't expect everyone to know everyone involved on a first name basis, immediately. We can't subject everyone to 21 questions about their personal and work history before responding to them, as that would also be rude. The replies in the first half hour all didn't have the context you had, and have now provided to the thread, and we universally reacted the same way. Mine more measure and polite, hence why its ranked highly and getting tons of replies now that activity has settled in.
And I'd be annoyed too, if I was working my ass off on a project, and continually seeing marketing measures being posted that I didn't think were valid.
But marketing lies (the engineering term) by figuring out when a lie isn't a lie (the legal term). It's literally their job. No news there.
The first 3 replies to Kenton, including mine, don't recognize Kenton immediately or on a first name basis just from their account name. So, we all reacted the way people react to the generic situation at hand: be careful, you're attaching your name and position to something that further commits your employer to a narrative you're trying to dispel.
I appreciate you providing this info, between that and Kenton's replies about themself (ex. about protos and cap'n proto) its clears that he has extensive experience in industry and at big companies, and certainly knows the tradeoff they're making. That's awesome, and different enough from the median situation to obviate early advice re: commenting on your employer.
Most days I wouldn't post a comment like the one above.
This was a bit of a "fuck it, just this once" moment and I'm not proud of it, but I stand behind what I said and don't feel the need to delete it.
But the title of the post validates his opinion and his claim about fastly and cold starts can literally be found anywhere in their blog and the compute@edge landing page.
I think it's a valid and valuable comment.
Sure felt like mud slinging to me. Just the whole list of complaints they had at the start of the post was hard to get through.
They could just tell us what they did differently and show the result.
5 microseconds is a round trip distance of ~500 m in optical fibre, and is hence totally irrelevant for 100% of CloudFlare's end-users... unless they happen to live in a 2-block radius of a Cloudflare PoP and have their ISP providing them with Internet connectivity with direct fibre from the same data centre...
Take some responsibility and ownership for the authority and soft power you wield as a very senior individual contributor.
You've been in this industry too long, and have risen too far to not feel a sense of personal investment and responsibility to 1) the benchmarks your company publishes about your product, and 2) the policies around public experimentation and transparency with that product.
Talk to your manager, talk to his manager, figure out who is responsible for both and FIX IT. You are the only major provider that has a DeWitt Clause. It's embarrassing.
So, I'm the tech lead for Cloudflare Workers. In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don't know anything about either beyond what's publicly visible.
But as long as we're already mud slinging, I'd like to take the opportunity to get a little something off my chest. Fastly has been trumpeting for years that Compute@Edge has 35 microsecond cold starts, or whatever, and repeatedly posting blog posts comparing that against 5 milliseconds for Workers, and implying that they are 150x faster. If you look at the details, it turns out that 35 microsecond time is actually how long they take to start a new request, given that the application is already loaded in memory. A hot start, not a cold start. Whereas Workers' 5ms includes time to load the application from disk (which is the biggest contributor to total time). Our hot start time is also a few microseconds, but that doesn't seem like an interesting number?
We never called this out, it didn't seem worth arguing over. But excuse me if I'm not impressed by claims of false comparisons...
On a serious note, I've been saying for decades that benchmarks are almost always meaningless, because different technologies will have different strengths and weaknesses, so you usually can't tell anything about how your use case will perform unless you actually test that use case. So, I would encourage everyone to run your own test and don't just go on other people's numbers. It's great that Fastly has opened up C@E for self-service testing so that people can actually try it out.