https://github.com/ebrake/pulumi-journaling

Hey HN! I created a fork of pulumi to explore how we can bring Pulumi's state-journaling architecture to DIY and self-hosted state backends. It's available at https://github.com/ebrake/pulumi-journaling

Background: Pulumi recently detailed their journaling architecture (https://www.pulumi.com/blog/journaling/), which changes how state snapshots are synced. Instead of uploading a massive full state snapshot on every operation (which gets sluggish for large stacks), it streams incremental state operations in parallel, achieving up to 20x speedups.

Journaling optimization has historically been tied to Pulumi Cloud. This repo brings similar state journaling and crash-recovery mechanics (like tracking epoch manifests) that works for file/S3/DIY backends.

I measured these timings against standard OSS pulumi using a GCS rapid bucket (https://docs.cloud.google.com/storage/docs/rapid/rapid-bucke...) which has low latency and is inexpensive given the amount of storage usage with a state bucket. Speedups become more effective the more resources you have.

Resources | Updated | Without journaling | With journaling | Speedup ----------+---------+--------------------+-----------------+-------- 24 | 15 | 19.66 | 1.69 | 11.6x 91 | 50 | 71.36 | 2.12 | 33.7x 152 | 50 | 70.97 | 1.91 | 37.1x

Would love feedback from anyone else working on IaC state management, concurrency, or crash recovery patterns!