stephenorr

I should probably also point out that the "improved" architecture turned out to be corrupting customer data for more than 6 months... which led to termination of that individual.

I'd already left by that point, after a heated conversation in which I was told "you're not half as good a software engineer as you think you are". Well, duh.

Amusingly, the PHP app is still running, still working.

The attitude might have been something to do with the individual's SRE background. In that context, I understand the desire to ship small changesets that have a limited blast radius.

It's just not really practical for actually building software.

I wish it had been satire. This was fundamentally one of the worst parts of my coding career, and it was disappointing because the company had been amazing to work for prior to this individual's arrival.

Post-arrival, I was ordered to rewrite a working, stable PHP web app into Go ("Google doesn't use PHP, because it doesn't scale"). We weren't allowed to write RESTful APIs any more, everything had to use Protobuf.

And there was a physical fight in the office one day between two of the developers, that culminated in one of them storming out and never coming back.

Well, things did get done, but not the right things.

The rule wasn't hard and fast, but you had to have a damn good reason why you wanted to add bigger commits. Protobuf definitions didn't count, so you could get away with doing longer commits for those.

Tests were discouraged anyway ("Google just tests in production", we were told), so there wasn't a lot of that sort of thing.

An ex-manager (ex-Google, too) insisted we move from GitLab to Gerrit, and mandated that all commits be a maximum of 5 lines, ideally fewer.

This led to a complete loss of all context; without that overview, you might very well be releasing code faster, but it could be the wrong solution to the original problem - and crucially, you won't know until it's out in the wild.

A preference for smaller, more focused PRs is fine - it certainly makes them easier to review independently - but I think putting any sort of limit on the size makes your team more likely to omit things like tests and also to not have sufficient context to understand the overall problem space.

It turned out that DO's base images for the worker nodes had automatic updates turned on; these were kicking in at around 6am each day, and causing the nodes to fail.

I'm sure their official incident report will have more details, but right now it looks like this is nothing to do with Kubernetes directly, but the underlying OS.

DO have now disabled those automatic updates so that this stops happening; it's been stable since then.

Still going on this morning.

If you're experiencing difficulties, the fix for us was to create a new node pool, then force kill the individual nodes in the previous pool. That forces k8s to move the workloads to the new pool.

You can't rely on the workloads moving themselves, because there's some weirdness around Cilium that's preventing them from being cleaned up properly. I'm not an expert on why, but the operator looks to be having trouble connecting to the Cilium daemon.

I've had to do this at least twice now, and DO Support don't seem to be any closer to a resolution. I've moved to a fixed-size node pool now to see if the cluster autoscaling is part of the problem.

There's been no real update from DO on this one.

Absolutely shocking to have an entire production-grade product broken for more than 4 hours with bugger all in the way of information about what's going on.

I have clients ready to move off Kubernetes entirely because I can't give them any idea when their sites will be back up and running.