We replicate sharded, versioned, csv-files on-demand from S3 to Cloudflare edge for code run by workers.dev.

Cache invalidation is tricky. Avoid it by ttl-expiring caches as frequently as budget allows. We, instead, employ a (probably over-engineered) hand-rolled set-reconciliation algorithm to match our read/write patterns.

do it! Years ago my team messed around with using lambdas to pull/push data from sqlite into s3 and it worked pretty well for certain regime sizes and workloads! Then we partitioned the data to filter out the slow zones into more robust/long running things like a job queue and postgres

Why not just include the sqlite DB as part of your lambda deployment package?

As long as you can fit it in your 1MB zip file, for viewer-request events.

For origin-requests you get 50MB. See https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

You only have five seconds to process the entire request. If your lambda has to make a request to download a sqlite database every time, and the process it, it probably needs to be small enough to fit into 1MB anyway.

AWS Lambdas can run up to 15 minutes actually

Lambda functions running @Edge have different limits compared to vanilla AWS Lambda. You get 30 seconds to respond if it's on a cache miss (Origin Request/Response), or 5 seconds if it's on a cache hit (Viewer Request/Response): https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

Lambda (at least the non-@edge version) comes with 512MiB of /tmp directory storage

And "The same Lambda execution environment may be reused by multiple Lambda invocations to optimize performance. The /tmp area is preserved for the lifetime of the execution environment and provides a transient cache for data between invocations. Each time a new execution environment is created, this area is deleted."

https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-...

https://aws.amazon.com/blogs/compute/choosing-between-aws-la...

I'm playing around with just that.

Regular Lambda (not edge) are just container images up to 10 GB in size.

There's also the layers thing. A function can use up to five layers at a time. The total unzipped size of the function and all layers can't exceed the unzipped deployment package size limit of 250 MB. Can update separately from the container, almost as fast as a regular fs.

And finally it has access to EFS which is faster than S3.