Of course there are always exceptions. Often stale data is ok and you can ship data snapshots to the edge so that they can be served without going back to the central DB. But in many cases some basic cache policies with a CDN can be nearly as effective.
Are you saying it would result in a better user experience when I only use host1 which is behind the CDN and add no-cache headers to the request that can't be cached?
We considered using Cloudflare Workers as a reverse proxy, and I did extensive testing of this (very reasonable) assumption. Turns out that when calling back to the origin from the edge, CF Workers established a new connection almost every time, and so had to pay the penalty of the TCP and TLS handshake on every request. That killed any performance gains, and was a deal breaker for us. It’s rather difficult to predict or monitor network/routing behavior when running on the edge.
Keep in mind that Cloudflare is a complex stack of proxies. When a worker performs a fetch(), that request has to pass through a few machines on Cloudflare's network before it can actually go to origin. E.g. to implement caching we need to go to the appropriate cache machine, and then to try to reuse connections we need to go to the appropriate egress machine. Point is, the connection to origin isn't literally coming from the machine that called fetch().
So if you call fetch() twice in a row, to the same hostname, does it reuse a connection? If everything were on a single machine, you'd expect so, yes! But in this complex proxy stack, stuff has to happen correctly for those two requests to end up back on the same machine at the other end in order to use the same connection.
Well, it looks like heuristics involved here aren't currently handling Workers requests the way they should. They are designed more around regular CDN requests (Workers shares the same egress path that regular non-Workers CDN requests use). In the standard CDN use case where you get a request from a user, possibly rewrite it in a Worker, then forward it to origin, you should be seeing connection reuse.
But, it looks like if you have a Worker that performs multiple fetch() requests to origin (e.g. not forwarding the user's requests, but making some API requests or something)... we're not hashing things correctly so that those fetches land on the same egress machine. So... you won't get connection reuse, unless of course you have enough traffic to light up all the egress machines.
I'm face-palming a bit here, and wondering why there hasn't been more noise about this. We'll fix it. Talk about low-hanging fruit...
(I'm the tech lead for Cloudflare Workers.)
(On a side note, enabling Argo Smart Routing will greatly increase the rate of connection reuse in general, even for traffic distributed around the world, as it causes requests to be routed within Cloudflare's network to the location closest to your origin. Also, even if the origin connections aren't reused, the RTT from Cloudflare to origin becomes much shorter, so connection setup becomes much less expensive. However, this is a paid feature.)
In my testing, the second fetch() call from a worker to the same origin ran over the same TCP connection 50% of the time and was much faster.
We want to use Workers as a reverse proxy - to pick up all HTTP requests globally and then route them to our backend. So our use-case is mostly one fetch() call (to the origin) per one incoming call. The issue is that incoming requests arrive to a ~random worker in the user's POP, and it looks like each Worker isolate has to re-establish its own TCP/TLS connection to our backend, which takes a long time (~90% of the time).
What I want is Hyperdrive for HTTPS connections. I tried connecting to backend via CF Tunnel, but that didn't make any difference. Our backend is accessible via AWS Global Accelerator, so Argo won't help much. The only thing that made a difference was pinning the Worker close to our backend - connections to the backend becamse fast(er) because the TLS roundtrip was faster, but that's not a great solution.
Again, origin connections are not owned by isolates -- there are proxies involved before we get to the origin connection. Requests from unrelated isolates can share a connection, if the are routed to the same egress point. Problem is that they apparently aren't being routed to the same point in your case. That could be for a number of reasons.
It sounds like the bug I found may not be the issue in your case (in fact it sounds like you explicitly aren't experiencing the bug, which is surprising, maybe I am misreading the code and there actually is no bug!).
But there are other challenges the heuristics are trying to solve for, so it's not quite as simple as "all requests to the same origin hostname should go through the same egress node"... like, many of our customers get way too much traffic for just one egress node (even per-colo), so we have to be smarter than that.
I pinged someone on the relevant team and it sounds like this is something they are actively improving.
> The only thing that made a difference was pinning the Worker close to our backend - connections to the backend becamse fast(er) because the TLS roundtrip was faster, but that's not a great solution.
Argo Smart Routing should have the same effect... it causes Cloudflare to make connections from a colo close to your backend, which means the TLS roundtrip is faster.
Cloudflare seems to consistently make all types of network improvements behind the scenes, so I’ll continue to monitor for this “connection reuse” feature. It might just show up announced.
none of this comment applies to back end + db in the same server or colo.
Yes, because that way you can leverage the CDN to defend against DDoS issues, and you can firewall the origin server itself so only the CDN is allowed to communicate with it, but no one else.
I've personally seen this with a game recently and my own ISP. I can ping their West Coast servers in 20-40ms while pinging their Chicago servers takes 200ms+.
This is actually one huge flaw with the current way IP is done. Because your ISP is closely tied with the last-mile of routing, you have very little control over optimizing those backend layers of routing.
This doesn't explain why you wouldn't just run a thin proxy at the edge and instead want to run a full PHP app. For that, my guess is this makes sense when you don't want to run a full server all the time or if you want it to be rapidly scalable.
Some other providers have tried to solve the problem with replicated databases that are used for local reads.
I can't go into details yet, but Wasmer will offer a solution for this problem quite soon as well, which hopefully will be much more seamless than existing strategies.
(note: I work at Wasmer)
Unless you host the database in the same network as your edge provider, appliation <-> database is where the bottleneck is.
Wordpress is a pretty bad example though, surely you’d just CDN cache the pages!
i'd like to see a comparison against a firecracker vm and a docker container
The builds are based on Dockerfiles, but for deployment we transparently convert that to unikernels.
For example, a form builder like typeform could cache form definitions on edge and render them close to users. Submissions would require db communication but the entire experience could be better.
Otherwise it is just not worth it.
BTW, I don’t even think PHP running slower in wasm would be that important. These things generally depend on IO performance rather than runtime if you are not doing a lot of calculations on the edge. WASM is also pretty fast these days so..
ironically everything is petty fast and everyone waste that speed adding things that allow for a cheaper labour. wasm included. it went from a "have some expertly optimized code in the browser" to "just compile this hide cpp thing and ship as a react component" to then "it allows our servers to run any crap without knowledgeable staff in each language"
we added so many Layers to cheapen labour that now all the labour costs goes into managing the Layers.
I agree with you, and it's not faster. With the risk of sounding old (get off my lawn): I assure you the megabytes of unneeded Javascript they're downloading to the client are ten times slower to execute than the milliseconds they may be saving on the wire.
You do caching so you don't always have to go back to the DB is the main thing. The other interesting capability is light scripting at the edge. Think "this page is cached but I'm going to add in a custom username".
There's some more exotic stuff around sharded database locality. But I'll skip over that because distributed mutable state is hard and its not my specialty.
Connection warming and shorter tls negotiation round trip times are also a thing but probably less important compared to the database thing you are mentioning.
But mostly this architecture makes sense if you can do some caching or if you're doing more data/log collection stuff.
For prior work see ESI (edge side includes). Assembling stuff at the edge does have some benefits but it has been hard to create a good interface for that.
Client talks to closest system, TLS gets terminated as soon as possible.
Certain things can be cached on edge which normally can't be cached just through basic http (like as soon as you are logged in).