I work in data storage. Am I the enemy, then? ;)
> can't even imagine managing hundreds of servers whose state is unpredictable at any moment
Be careful not to conflate immutability with predictability. The state of these servers is predictable. All of the information necessary to reconstruct them is on a single continuous timeline in source control. But that doesn't mean they're immutable because the head of that timeline is moving very rapidly.
> can't be terminated and replaced with a fresh instance for fear of losing something.
No, there's (almost) no danger of losing any data because everything's erasure-coded at a level of redundancy that most people find surprising until they learn the reasons (e.g. large-scale electrical outages). But there's definitely a danger of losing availability. You can't just cold-restart a whole service that's running on thousands of hosts and being used continuously by even more thousands without a lot of screaming. Rolling changes are an absolute requirement. Some take minutes. Some take hours. Some take days. Many of these services have run continuously for years, barely resembling the code or config they had when they first started, and their users wouldn't have it any other way. It might be hard to imagine, but it's an every-day reality for my team.
You’re the prison guard.
I don't trust predictability. Drift is always a nightmare. Nothing is ever as predictable as you would like it to be.
>You can't just cold-restart a whole service that's running on thousands of hosts and being used continuously by even more thousands without a lot of screaming.
If it's architected well you can :)
Except that state and its manipulation is usually the primary value in technology.
> I can't even imagine managing hundreds of servers whose state is unpredictable at any moment and they can't be terminated and replaced with a fresh instance for fear of losing something.
Yes, that sounds awful. That's why we have backups and, if necessary, redundancy and high availability.
Exactly and thats why you put state in data stores and keep your servers immutable.
State is always the enemy in technology.
I can't even imagine managing hundreds of servers whose state is unpredictable at any moment and they can't be terminated and replaced with a fresh instance for fear of losing something.