Yeah, while I would have tried to investigate this a bit, this seems to be an issue on Cloudflare

Bypassing the rules is a workaround, but not a fix

This jumped out at me too. Cloudflare keeps many diagnostic and debugging tools available only on the Enterprise plan (typically >$3k per month). When something like this happens to most small teams and startups, they are working in the dark even on the Business plan at $200/mo.

FWIW CloudFlare has worked with us time and time again to get pricing in line with our expectations. We have numerous "Business Plus" or "Free Plus" plans across various accounts and zones that include features that are, as listed in the marketing, only available to Enterprise.

Just reach out to someone at Cloudflare. I'm sure they'd love your business.

And if you have to disable WAF, what exactly is Cloudflare doing for you?

Would it still do DDOS even without the WAF?

Our DDoS mitigation is separate yes and will still work.

Edge Cache with unmetered egress?

unmetered up to a point, but yeah. Also network-level DDoS protection

If you are reaching the point where they are sending you emails asking about what you are doing, you should be paying for it.

yeah, but sadly there aren’t many CDNs that offer WAF, even the most basic one. I literally begged bunnyCDN to build one so we can switch from CF. It’s on their roadmap for like forever.

In the article, the author disables the WAF only for Stripe outbound IPs, which can be presumed to be safe (unless Stripe's machines/IP space gets hacked). The WAF still works for traffic from all other IPs

Even with the WAF disabled (at least as much as I can disable it without the Enterprise plan, i.e. "Essentially Off"), I've found it will still block legitimate requests. Tainted CGNAT or dynamic IPs are my guess.

The WAF doesn't really matter for my use case as the route is handled by a CF worker, in fact I'd prefer it doesn't get in the way.

And WAF isn't the only design option for webhooks. WAF = define what to block. Another options is to define what to accept (block all else by default).

OpenZiti approach (1) for example:

a. enroll each side of the webhook w/ X.509 identity

b. X.509 gates a network overlay between the servers

c. each server initiates outbound sessions to the overlay

d. block everything else (deny-all inbound on both servers)

(1) disclosure: i am a maintainer of the openziti foss, and you can only (fully) use the technique above if you have enough control of both sides, e.g. use a Lambda function: https://blog.openziti.io/my-intern-assignment-call-a-dark-we...

Mutual TLS auth seems like such a no-brainer for APIs like these.

I do wonder, though, how does OpenZiti deal with client certificates expiring? Do you not set/enforce an expiration date or do you have some kind of automated renewal mechanism?

Cert renewal is why it's not used everywhere, I think. While OAuth2 / client credentials is complicated, at least .jwks endpoints make cert rotation easy.

yes, can be tricky, as you state. see above for the openziti approach.

your overlay network fabric infrastructure - ziti edge routers - leverage the ziti edge client API to submit CSRs for new certs x days before expiry.

for client endpoints, there are currently 2 choices and at least 3 in the future:

1. admins chooses to enable client endpoints to continue to auth with expired client certs. the admin revokes access (when necessary) at an endpoint level or at a service level (least privileged access) via those constructs, rather than cert constructs.

2. admins don't permit expired certs to be used. the admin is managing the certs.

3. as a third choice for client endpoints, ziti has plans to enable admins to use the API used by the fabric infra mentioned above. this is future, with timing mainly dependent on openziti community priorities and related work.

The problem is that you hope this doesn't encourage the developers to lapse on RBAC and let an authenticated client do everything.

absolutely. in the case of openziti, the x.509 is the first authentication gate (optionally paired with ziti posture checks and mfa). everything from there is least privileged, attribute-based access, at a service level. default is closed - access to no services (even if authenticated at the level of the private ziti network fabric overlay).