> We spent months building custom caching for our GraphQL API from scratch

Could you not abandon GraphQL? Returning non-customized responses, while taking more bandwidth, is much more cache friendly for the client, proxies and particularly servers - where you can do simple yet powerful things (e.g. version-based invalidation and pre-generated payloads).

Like, I went to https://spectrum.chat/explore and clicked on "Tech. It uses GraphQL to load the communities.

Why not just hit /v1/communities?category=tech

which would loosely translate into one of:

-- if you want to serialize the object on each get

select * from communities where tags @> array['tech']

-- if you can look up the communities in a cache by id

select id from communities where tags @> array['tech']

-- if you have a high read / write and can just pre-serialize the payload and then glue together the response

select summary from communities where tags @> array['tech']

I think it depends what you want as your cache invalidation policy. Simple TTL-based cache invalidation wouldn't necessarily need GraphCDN. However, no matter if REST, GraphQL or any other protocol - making sure, that when content changed, to invalidate it properly on the edge is not trivial.

If I understand correctly, you're suggesting to cache on a data-layer level (below the app in the stack) instead of what we do - above the app in the stack - just in front of the client.

That is also a valid approach. It'll need more custom code in your application and has the disadvantage, that your origin will still be hit on every request, while we fully cache on the edge - cached queries won't hit your server anymore and will take load off origin and have the minimum latency possible.

I'm suggesting that you can cache both at the data layer and "above the app". I was just highlighting one particularly powerful patterns available if you're having trouble scaling (like my parent said).

> making sure, that when content changed, to invalidate it properly on the edge is not trivial

The trivial way to do this is simply not to invalidate. Include a version in the cache key. When content changes, the version increases, and clients get new versions. (I think this is sometimes called lazy cache invalidation). It works well with LRU caches. It's still a hit for the top-level keys, but you avoid all the heavy data load and rendering.

Granted, that doesn't solve every case, but I'm not sure it's fair to say that it'll take more custom code when the initial approach took "months building custom caching" that never really worked. Also, I think you'll find your origin under less load due to having fewer variants.

That makes sense! I like the versioning approach - of course requires apps to have a version for their entities, but if they have, that's indeed a great way to leverage it. And yes, that also sounds like a solution that wouldn't take months to build.

Or use something like Odata that solves the same problems as GraphQL (returning complete and tailored trees of objects in one go), but uses regular GET operation to it's still compatible with normal http caching?

Many production GraphQL apps these days use persisted queries [0] via GET requests. So the HTTP caching itself is not an issue in GraphQL anymore.

[0] https://blog.codecentric.de/en/2020/05/how-to-secure-a-graph...

Max & Tim, thanks for building this. This is something I recently spent 2 weeks trying to solve, then more pressing issues came up so we threw more CPUs at the problem. So from that I can appreciate how complex this is to solve, and the UX is awesome, so cool that you can edit the cache keys in the UI.

Do you/will you handle caching individual request layers? For example if I need part of my query to always be fresh, but some expensive part can be cached for hours, is this possible? And somewhat related, what about keying on operation variables?

Thank you for the nice words, glad to hear you like GraphCDN.

We don't currently cache partial queries / request layers. For now, I would recommend splitting the query into two requests — one that loads the uncached data and one for the cached data.

We're definitely thinking about this exact use case though, stay tuned!

Ah yes, the devil in the details.

Partial query caching appears to be the unicorn we all would like to chase, capture, and ultimately study.

GraphQL is powerful, But with great power, comes great responsibility, and it is too easy to dig your own grave if blindly jumping in with it.

I wonder if we can learn a thing or two, or just flat out steal, some of the concepts used in flagship relational databases. For example, how Postgres has a query optimizer tucked away secretly under the hood that attempts to alter queries to be more efficient.

This is a super interesting project and thanks for posting it.

I'm just wondering why you decided on Fastly over Cloudflare Workers or AWS Lambda Edge for this?

Good question! These are the main reasons:

Fastly has much faster cache purging - it can purge any content globally in about 150ms. Purging is one of THE crucial parts which make GraphCDN work. With such a fast purging, we can deliver Read-after-write consistency . In our tests with Cloudflare that took a few seconds.

Additionally, in our tests, Fastly was in general just a bit faster.

While we're also JavaScript and V8 fans, our edge layer on Fastly is written in Rust - which is a pleasure to use, especially for a product as ours. Fastly runs it as WASM on the edge - they can load the worker in about 40 microseconds.

Fastly Compute@Edge is still in limited availability, but if you get a chance, I highly recommend checking it out!

> it can purge any content globally in about 150ms

Hmm, how can that be? Light in a straight-line fiber optic cable would take 200ms to traverse a greater circle around the world. Add some bends, relays, routers, not to mention servers, and it'll only get longer...

Are you sure the content is really being purged globally in that time, and not just from your local point of presence?

(Disclosure: I'm an engineer on Cloudflare Workers. I have no idea how Fastly does purges. Just trying to understand what you mean here...)

What do you mean by greater circle? The circumference of the earth is 40,075,000 m. Speed of light is 299,792,458 ms. So we're talking about 40,075,000 / 299,792,458 = 0.1337 -> 134ms to get around the earth once. As the earth is an ellipsoid that is mostly perfectly round, you can assume, that from any point A to B you'll just need to travel half of that, so we're at 67ms for light speed.

Looking at the 150ms now, which is about 2.3 times the 67ms, this kinda seems reasonable.

And yes, this is global purging - I just reconfirmed it with someone from Fastly.

That's the speed of light in a vacuum. The speed of light in fiber optic is slower.

Travelling half of the distance is not good enough, because if you haven't received _confirmation_ of the purge, then you can't trust that it really happened. There could be network errors, etc.

Sorry but it's not physically possible to do a global purge in 150ms.

> we can deliver Read-after-write consistency

Based on the comment on max below that's not read-after-write consistency but it just becomes eventually consistent but ideally within ~150ms. What would be the consequences of bypassing the cache for read after write, does that break your model in any way? (I guess at the very least you will lose your statistics feature)

In all mutations you run through GraphCDN, we wait until the purging is done (as you said, ideally ~150ms). So that means at the point the mutation result comes back to the client, all other queries will get the new content from then on.

You can bypass the cache if you want to, but it's not necessary. As it would just be like a Cache MISS, that would totally work.

I suppose that means writes slow down with the number of dependencies/ dependent caches? How does one estimate the added cost of that write?

Well, that depends on the purging characteristics of Fastly - this blog post is quite nice to read https://www.fastly.com/blog/building-fast-and-reliable-purgi...

We have a limit - you can only "tag" up to 1k items within one query. If you have more items in there, we can't purge all of them.

Due to the nature of the broadcast protocol of Fastlys purging implementation, I assume that it scales quite well - they have really big customers using this for a few years already.

There is a big difference if you're using fastly's cache purging for simple cache invalidation without consistency requirements or if you are trying to do something like you're building.

We're using Fastly ourselves and I was not aware up until today (and in fact I'm taking your word for it, since I can't find it in the docs) that fastly provides consistency guarantees for purges.

Man, we had this exact use case and a similar solution designed but we weren't able to get access to Fastly's new edge compute.

Now you can just use GraphCDN and don't have to build it yourself! ;)

This was at a fairly large enterprise that utilized Fastly, and has since switched to Akamai. We had many meetings with our AM and their architects to get access to their compute engine over multiple years but never could get them to open the doors. If I could migrate our GraphQL services to a new domain, I'd certainly try it out!

Good work!

What was it about this problem that made you need Compute@Edge rather than the standard Fastly/Varnish VCL-backed caching? If you tried that route, how far were you able to get with Varnish before needing the new Compute@Edge product?

Good question pbowyer! We actually do a bunch of GraphQL parsing / calculations in the Compute@Edge layer. So without that, we would not be able to provide our current features.

We also support a feature we call "scopes" - where we both support cookies and headers in the "Vary" header so you can cache user-specific data only for the user with a specific "Authorization" header for example.

In order to support scopes with cookies, we needed Compute@Edge and could not make that work with VCL on its own. But believe me - we tried. Our first version was actually mostly built on top of VCL.

It's been awhile but IIRC you can't inspect the body on a POST request using Varnish. Either that or any real compute in Varnish gets super gnarly. I also think Fastly has a distributed K/V for state storage that you can access if you're using their new compute.

That is correct, you need to first transform the POST request into something Fastly can cache. They don't have a K/V Store besides the edge dictionaries with max 8kb value size yet, but I've heard something is coming...

Ah cool, thanks for the update. I haven't looked at their stuff in over a year. Congrats on the launch, looks really slick.

Fastly has have trouble caching POST requests because Clustering [0] doesn't work with POST requests (AFAIK there is not workaround).

Does Compute@Edge have this limitation?

[0]: https://developer.fastly.com/learning/vcl/clustering/

Under the hood, we're turning the POST requests into get requests, so that limitation shouldn't affect us. As far as I know, underlying C@E behaves like the one used in a VCL service.

What happens when the query string gets too long?

The question that had to be asked :D First of all, with the current query string size, you can already send quite big queries (8kb), which should suffice for many use-cases. The introspection query, already quite huge for reference is 1kb. For the ones who need more - splitting the values into `Vary` headers is an option to further increase it. At some point - around 50kb there just is a hard limit. If you then want to send an even bigger query - we have persisted queries on our roadmap. Then you'd just send a hash.

How does the invalidation work if data changes independently in the backend, i.e. without using mutations?

We give you a purging API (GraphQL as well) where you can target specific queries (`allUsers`) or types (`User`) that you want to purge. You can read more here: https://graphcdn.io/docs/cache-purging

How GraphQL subscriptions are handled?

Are they supported?

Do they count as a single or multiple requests for pricing purposes?

As of now, GraphCDN only caches queries. Subscriptions are just passed through to the origin - a normal WS connection is built. One WS connection counts as one request.

Really interesting. Can this work with Hasura?

Yes, it does! We have more and more people who want a more powerful cache setup than what Hasura offers or are just self-hosting Hasura. Both self-hosted and Hasura cloud work!

Shut up and take my money! This will save me a bunch of time from my original plan of rolling a distributed Elixir proxy caching cluster to put in front of Hasura.

I'd love to see a write up/blog post for working specifically with Hasura and their auth system. Would probably be helpful for seo purposes as well.

Absolutely, we'll work on that. Let us know if you have any questions or run into any trouble at [email protected]!