News for hackers. Stuff that matters.
Re:Ask HN: Has anyone migrated off containerized infrastruct... by imdsm on Friday August 14, 2020 @04:32AM (#24155350)
> otherwise your 3 hour code review, build, q/a, deploy pipeline will cost the company $180,000

Taking code review away, because we all know we're not going to sit around waiting for a review while a patch needs to go out urgently, if your build, test, and deploy pipelines take three hours then you have some serious problems that you need to address, and containers aren't it.

There are methods for handling hotfixes/patches to production quickly that work well in high volume sales website setups.

Re:Ask HN: Has anyone migrated off containerized infrastruct... by mumblemumble on Friday August 14, 2020 @08:31AM (#24157023)
> Taking code review away, because we all know we're not going to sit around waiting for a review while a patch needs to go out urgently

I worked at a trading firm where you could add a zero to that cost for a 3-hour outage, and I can tell you that the one thing we would absolutely never skimp on was code review. Because the cost of a bad "fix" that actually makes things worse has the potential to be greater still, and because humans are most likely to make silly mistakes when they're working under intense pressure.

What we would do instead is slightly intensify the code review process by pair programming the hotfix, and ensuring that a third developer who was familiar with the system in question was standing by to follow up with an immediate review.

Re:Ask HN: Has anyone migrated off containerized infrastruct... by allannienhuis on Friday August 14, 2020 @10:59AM (#24158761)
Pair programming anything as urgent as a hotfix works really well. It takes some pressure off of the developer working on it and turns it into a team event.

We will even sometimes keep the video call open until deployment is done and production validation is complete - the devops guys get the information they need, someone else is keeping an eye on the checklist and calling out items if necessary, etc.

Re:Ask HN: Has anyone migrated off containerized infrastruct... by mumblemumble on Friday August 14, 2020 @11:57AM (#24159595)
Oh yeah, absolutely. I was happiest when I also had the attending ops person looking over my shoulder while I worked on the fix.
Re:Ask HN: Has anyone migrated off containerized infrastruct... by codemonkey-zeta on Friday August 14, 2020 @08:58AM (#24157290)
I really like that approach. It also takes pressure off the tech lead or whoever is implementing the fix, and transforms the patch into a full-team responsibility. I bet this sort of behavior makes for strong and effective teams.
Re:Ask HN: Has anyone migrated off containerized infrastruct... by pdimitar on Friday August 14, 2020 @05:40AM (#24155787)
Please do tell. Our containers never take less than 10 minutes to deploy (a single one).
Re:Ask HN: Has anyone migrated off containerized infrastruct... by owenmarshall on Friday August 14, 2020 @07:59AM (#24156739)
I have app stacks that deploy a dozen containers in seconds because they are stateless and close to "functional" – just transformations over inputs.

I have app stacks that deploy a dozen containers over an hour because the orchestration takes time: signal the old containers to drain, pause for an app with a very long initialization time to settle, gradually roll traffic to the new one to let caches warm, and then repeat.

In both of these cases, deployment is a function of the application. There's nothing infrastructural that puts a time floor on things.

Re:Ask HN: Has anyone migrated off containerized infrastruct... by pdimitar on Friday August 14, 2020 @08:02AM (#24156760)
Sure, not contending that. It's just that in my memories blue/green deployments still took less time, although I can't say how much.
Re:Ask HN: Has anyone migrated off containerized infrastruct... by ryanjshaw on Friday August 14, 2020 @08:16AM (#24156873)
Not a serious reply, but it may interesting: One million containers over 3500 nodes in 2 minutes: https://channel9.msdn.com/Events/Ignite/Microsoft-Ignite-Orl...
Re:Ask HN: Has anyone migrated off containerized infrastruct... by asdfaoeu on Friday August 14, 2020 @07:31AM (#24156547)
Well 10 minutes is way less than 3 hours.
Re:Ask HN: Has anyone migrated off containerized infrastruct... by pdimitar on Friday August 14, 2020 @07:56AM (#24156706)
Certainly. But it does add up and it does kill motivation for rapid iteration. :(
Re:Ask HN: Has anyone migrated off containerized infrastruct... by jonpurdy on Friday August 14, 2020 @09:16AM (#24157500)
10 minutes to deploy to production should be fine, but rapid iteration shouldn't be happening in production. It sounds more like the complaint is related to running a similar setup in dev, and taking 10 minutes to see changes during development (which is understandably too long).
Re:Ask HN: Has anyone migrated off containerized infrastruct... by pdimitar on Friday August 14, 2020 @11:23AM (#24159087)
Yep, that's what I meant. I have no choice but to run a local k8s cluster or else I can't test.
Re:Ask HN: Has anyone migrated off containerized infrastruct... by Guthur on Friday August 14, 2020 @08:27AM (#24156997)
and then we're back to the original statement, if you're rapidly iterating hot fixes you have serious problems and likely doing it wrong.