News for hackers. Stuff that matters.
Re:Prime Video Abandons Serverless Distributed Stack for Mon... by withinboredom on Thursday May 04, 2023 @04:02AM (#35812573)
I'd ask why people are merging things that break the tests? I worked on a monolith with hundreds of devs and I can count the time the tests failed because someone force-merged something in an emergency on one hand. It was generally unacceptable to merge something when tests failed; you had to have a really good reason.
Re:Prime Video Abandons Serverless Distributed Stack for Mon... by anonzzzies on Thursday May 04, 2023 @05:37AM (#35813070)
Yes, this seems weird; merging breaking code is not an option. The 'breaking team' will have to wait/fix on their side, not us waiting on them for deployment of our working and tested features.
Re:Prime Video Abandons Serverless Distributed Stack for Mon... by withinboredom on Thursday May 04, 2023 @05:50AM (#35813126)
There are some extreme circumstances where pushing broken tests to production make sense. For example, if you push a simple change to simply `return false` and disable a feature in code. In this case, the tests using it will probably fail but the desired behavior happens in production. At this point, you have a bit more time to set the tests to 'skip' while the load is shed in production. Even if you break tests on purpose, you should fix them asap as you are blocking literally every other team in the company. Thus, you need a really good reason to do so (like if you didn't do it asap, global downtime would ensue).
Re:Prime Video Abandons Serverless Distributed Stack for Mon... by Akronymus on Thursday May 04, 2023 @06:06AM (#35813231)
Currently we have the problem of merges of two working branches occasionally resulting in a broken one. How does one solve that?
Re:Prime Video Abandons Serverless Distributed Stack for Mon... by withinboredom on Thursday May 04, 2023 @06:35AM (#35813408)
Don’t merge. Rebase only. Keep a linear history.

When committing, do a ff-only of ‘main’ to your branch. Yes, this forces everyone to rebase before “merging” but in practice, this results in the least amount of failures, tests being run after you resolved any conflicts, etc.

If you can use GitHub merge queues, that solves a ton of this, and you can run tests on the final merge before actually merging instead of relying on rebasing.

Re:Prime Video Abandons Serverless Distributed Stack for Mon... by whstl on Thursday May 04, 2023 @09:12AM (#35814726)
"Don’t merge. Rebase only. Keep a linear history."

This. It makes life so much simpler. With teams that don't have a lot of experience with git, however, I tend to use the "Squash and Merge" feature, coupled with forcing a linear history.

Re:Prime Video Abandons Serverless Distributed Stack for Mon... by lxgr on Thursday May 04, 2023 @09:18AM (#35814808)
Requiring a passing integration branch before merging to master. Merging the integration branch then becomes a fast-forward merge.

Alternatively, if you have a low enough merge volume, requiring mergers (by policy) to squash and rebase (and re-run tests before attempting to merge) can work too, as others have already mentioned.

Re:Prime Video Abandons Serverless Distributed Stack for Mon... by jmillikin on Thursday May 04, 2023 @06:01AM (#35813196)
1. Add a test with a time-bomb (such as a test certificate with 365-day duration), wait a year, and now your test fails without having changed.

2. Add a test with a network dependency, and when that dependency is slow / down / turned off, the test starts failing.

3. Add a dependency on a third-party Github repo that clones from `main`, and the next time some dev touches a file in that repo your test starts failing.

4. Add a test that allocates memory in proportion to size of the codebase (e.g. because it tries to build a giant in-memory tarball of all the .mp4 assets). Eventually it will get flaky when it starts scraping up against the build machine's limit. Extra fun if your builds run without defined memory limits on machines of different sizes.

In a monolithic build, there's all sorts of ways for a single person to cause other teams' tests to fail, even months or years after they've left the company. Some of them can be prevented mechanically (such as by running tests without network access), but a lot come down to "tell them to stop doing that".

That's why big companies never run one build per repo.

Re:Prime Video Abandons Serverless Distributed Stack for Mon... by withinboredom on Thursday May 04, 2023 @06:29AM (#35813375)
Hmm. We never had those specific issues. For (1), we had time bombs for sure, but those usually highlighted coverage issues. Dev ops would disable your test and tell you to fix your code.

IRT (2), network dependencies were forbidden in-general. Over any long enough timespan, the rate of failure is 100%. If you wanted to use the network, you had to consider the failure case and handle it in your tests.

For (3), all dependencies were committed as part of the repo. All dependencies had to be reviewed for any issues before being used, so this made sense. You simply weren’t allowed to just randomly include a new dependency without a review/PR to add it.

For (4), our dev environments had less memory than build machines and the same as production. If you couldn’t build it in a dev environment, it wasn’t getting committed without special treatment from dev ops (and a really good reason).

Re:Prime Video Abandons Serverless Distributed Stack for Mon... by jmillikin on Thursday May 04, 2023 @08:38AM (#35814321)
Yeah, all of those mitigations don't work as well when you've got thousands of engineers whose work would be blocked if some intern's badly written test blows up.

A monolithic build means that your ability to develop and deploy your team's code is dependent on every other team. As the number of teams gets larger, that multiplier really hurts.

Re:Prime Video Abandons Serverless Distributed Stack for Mon... by withinboredom on Thursday May 04, 2023 @10:41AM (#35816118)
It's a learning experience. If everyone learns from it, it probably (most likely) won't happen again. Everyone learns how to write better tests. And, like I said, if you absolutely need to merge something right this exact second and it can't wait until someone disables the failing test (or you can't do it yourself for some reason), you can always merge it even with failing tests.
Re:Prime Video Abandons Serverless Distributed Stack for Mon... by __turbobrew__ on Thursday May 04, 2023 @12:57PM (#35818552)
Average tenure at most companies is 2/3 years. You cannot rely on people learning from mistakes because there are always new people, you need to make it so they cannot make mistakes or if they make a mistake it isn’t going to block the whole company from executing.
Re:Prime Video Abandons Serverless Distributed Stack for Mon... by withinboredom on Thursday May 04, 2023 @01:05PM (#35818693)
Organizations learn, it gets embedded in the culture, tooling, and automation. I shared some of the rules that were embedded in our organization via style guides and onboarding.

There were a thousand automated checks to prevent you from doing the same thing as someone else that caused downtime in the past. It was virtually impossible to commit code that deleted/truncated a table, for example.