You don't even need to set it up by hand. Cache Reserve is the way to put Cloudflare's CDN with a persistent cache in front of your origin (which could be an S3 bucket). It's a 1 button config thing.

If you use Worker bindings, then there's no write-enabled keys. It's a fair critique obviously and the team is aware of some of the product gaps. We only just launched GA :).