No, R2 is not an "S3 cache". It's a replacement for S3 with no egress fees.
Yeah, don't use Sippy. It just stopped working after 2 weeks on multiple buckets and support came back with "have you tried turning it off and on again". It's not production ready.
Also R2 is extremely basic compared to S3. For example only trivial permissions are supported and auth tokens are bound to an actual account.
edit: apparently not writethrough: https://developers.cloudflare.com/r2/data-migration/sippy/