Something like Packer?
You are right that there are hidden pitfalls. But they shouldn't be a great deal if servers are cattle. A lot of corner cases are related to trying to manage servers that live forever. This shouldn't really be done in 2021 except in rare cases.
> i'm imagining a system where the local filesystem is more or less immutable and work is saved to separate filesystem
CoreOS worked like that. Immutable filesystem for the system itself. It also had a secondary "mirror" of this filesystem that was used for upgrades (and as a fallback in case boot failed). New stuff would either come when the instance first booted up (cloud-init), or via system upgrades.
They were acquired and so my experience with them ended there.
There's also https://nixos.org/
a better alternative would instead be something akin to Dockerfiles where you assert what packages you want to be installed on top of a base system, and if you want anything changed you update the dockerfile and rebuild the whole shebang.
i'm imagining a system where the local filesystem is more or less immutable and work is saved to separate filesystem. but having worked from computer labs in schools, i know how less than ideal that is when i need to use a program that isn't preinstalled to do my work, so maybe this all doesn't work and saltstack is a necessary evil