Yeah no thanks. It's probably better than completely random but software should be predictable and unsurprising.
I think it is a good reminder that most things you think of as being unchanging that are also directly related to a person.. aren't unchanging. Or at least any conceivable attribute probably has some compelling reason why some one will need to change it.
https://m.youtube.com/watch?v=r-TLSBdHe1A&t=14m10s
Discussing a performance regression due to longer username due to username being in ENVIRONMENT variable which changes memory layout of process.
It worked quite well.
Will it cost an extra lookup? It's cheap, and if you really need to, you could embed the lookup in some encrypted cookie so you can verify you approved some name->id mapping recently without doing a lookup.
Clearly, making sure that 1% of all teams gets fired for being unable to run unit tests, then slowly ramping that by a few percent each review cycle is a good strategy.
Ideally, the probability of breaking would drop off exponentially as you moved up the org chart. Something like "p ^ 1/hops_to_director_of_engineering" would work well. The trick would be getting the dependency to query ldap without being detected...
I was on a team where app updates were deployed using a canary system. A small percentage of users (say, 1%) received the update first, then the team watched for incoming crash reports from that cohort. If it looked good, the feature was rolled out to a few more people, and this was repeated. This allows you to identify a problem by only negatively impacting a relatively small percentage of customers.
The problem occurs when the calculation to determine which cohort the user belongs to is deterministic. In this case, the calculation was based on the internal ID of the user. This means some users always get the updates first, and deal with bugs more frequently than other users. Conversely, some users are so high in the list that they virtually never get an update until it's been tested by a wide user base, so their experience is consistently stable.
Or you might have a problem where some players in a video game consistently take more damage than their friends: https://news.ycombinator.com/item?id=34742505
Wow. I feel like this dependency should be named and shamed.
If done in a company with a monorepo I'd be especially interested in hearing more
Are there any big companies left which haven't adopted a monorepo?
I disliked how needlessly convoluted the pipelines are, and how some person pushing on accident to mainline can break everything.
So many things seem to be done the hard way.
Sure, but I bet that it was helpful to have the `bats` tool so that you could replicate the build locally, right? As compared with other build systems where (so far as I can see - though I may be wrong) you basically have to push a debugging change for replication.
> I disliked how needlessly convoluted the pipelines are, and how some person pushing on accident to mainline can break everything.
This is true of any CI/CD system, though? In any system, if there's no push-protection set up so that you can only merge into main(/line) once a change has been reviewed (and run your tests at the point of review so that you know the merge won't break anything), you have only yourself to blame for breakage.
> So many things seem to be done the hard way.
Genuine question - what do you find convoluted/hard about them? To me, the apparently-industry-standard of "push a change to your App Code, which triggers a build to generate a docker image, then trigger an automatic commit containing that Docker image to a Deployment Package, which is picked up by your CD system and creates a deployment" is way more convoluted. Having a conceptual "pipeline" built out of lots of little disconnected GitHub Actions (or whatever) is also way harder for me to wrap my head around than the CDK definition of a linear pipeline.
I figured a work around that involved having a separate manifest for the deps and packing them manually. It worked… I also tried the lambda without any dependencies and they the lambda’s dependencies were available in the instance even though they were not listed anywhere.
The needlessly convoluted part is getting a NodeJS function into prod, the forced change caused something to break even though I was already on 18_x
But I will not lie, bats was super useful when debugging another engineer’s build, and avoiding the whole push to debug is so so so so helpful.
For the pipeline thing, idk why it was setup as such; but it certainly broke everything once a push to mainline was done, kinda like a runtime error when it should have been a compiler error. Though it was certainly on us.
CDK is fine too. Tbh I kinda love CDK and don’t want to go to anything else when it comes to cloud deployment.
In retrospect, perhaps that one sour experience had just too much of an impact.
I want a better way to manage external dependencies mostly. Afaict for scala I will need to pull the package into Brazil to get it to work. An analogue to NPMPM would be great but I can see why there isn’t one yet. A colleague had issues getting python dependencies to work so yeah.
Here’s to hoping Peru would fix those.
On that point, I'm totally with you. The excessive caution about the software supply chain is probably justified given the impact of a potential incident, but certainly frustrating for the >99% of times that dependencies are safe.
However, it's not surprising when you consider the massive breadth of software that Microsoft builds, as one of the oldest and largest software development orgs.
I doubt single digit percentages of, say, the fortune 500 use them, and I'd be surprised if double digit percentages of companies use them at IPO.
As with anything, striking the balance and finding when to use a tool is the hardest question.
FWIW, I think best practice here is to hardcode all feature flags to off in the integration test suite, unless explicitly overwritten in a test. Otherwise you risk exactly these sorts of heisenbugs.
At a BigCo that’s probably going to require coordinating with an internal tools team, but worth getting it on their backlog. All tests should be as deterministic as possible, and this goes double for integration tests that can flake for reasons outside of the code.
Yes it's kinda fractal of bad practices that have to align for this problem to occur, but that's the nature of tech debt.
He’s not saying to alter any of the feature flags used for the test, but simply to record which were used during the test.
Simply logging doesn’t introduce any of the issues you are describing.
> I think best practice here is to hardcode all feature flags to off in the integration test suite
That's pretty clearly about forcing the flags to be off, i.e. altering them, and not about logging their values.
If you’re testing a new feature, you should have explicit tests for the enabled state (along with existing tests for the disabled state).
If you have bugs propagating up the stack from flags changing in low-level dependencies, the change to the dependency is probably not properly tested.
Alternatively, if the feature flag gates a change to the interface of the dependency, you should have explicit integration tests covering the systems on both sides of the change.
One way I have seen this handled is to restrict rollouts of a feature flag to 95% at most. That way turning a feature all the way on requires removing a flag from your codebase. It’s draconian, but honestly anything less than that leads to the situation you describe.
That way when you get a failure, you can reproduce it. And then one of the easy things to do is test which features may have contributed to it.
Nothing kills confidence in testing more than test flakes. It’s a huge drain on velocity and morale, and encourages devs not to trust test output.
If you want to have some sort of chaos monkey process that runs your test suite flipping feature flags at random and notifying teams of failures (along with some sort of resourcing to investigate) I could get behind that. But that should be something outside of the main suite that gates code deployment.
If a test passes when run by a dev pre-commit, it should pass in CI.
In the end, I think the real problem is that you can't test all combinations of experiments. I don't trust "all off" or "all on" testing. In my book, you should indeed sample from the true distribution of experiments that real users see. Yes, you get flaky tests, but you also actually test what matters most, i.e. what users will - statistically - see.
Basically, if you have N different features (let's assume they are all on/off switches, but it works for multi-values too), in theory you'd need to run 2^N tests to cover them all, which would become completely impractical. But, you can generate a far, far smaller set of test setups that guarantee that every pair of features gets tested together. Run those tests and you'll probably encounter most feature-interaction bugs in a much quicker time.
Let's say you have a bug you suspect is from an interaction of any one pair of 10 features being "on" or "off", but you don't know which specific pair causes the problem. Encode each of the states you could set up your code by a 10-digit binary string: 0000000000, 0000000001, 0000000010, 0000000011, etc.
We could try the 45 possibilities in some order, and we would expect that on average it'd take us 22.5 tries to find the bug. But notice how your "target set" is smaller than the universe of strings: there's only 45 pairs of features, but 1024 strings.
What happens if we try a random string of ones and zeros? Now, instead of catching just one possible pair, we are covering many pairs. The only problem is that we now won't be able to know exactly which pair caused the problem when it does. But we can build a corpus of strings that don't trigger the error vs. strings that trigger the error, and a random sampling soon converges on the correct pair.
If you think about why this works, it's because any of these random strings has about a 1/4 chance to trigger the bug: wlog we can reorder the bits so that the buggy feature are the first two digits, and then we see that we have a 1/4 chance of hitting "11" on those two digits.
The problem is that as you increase the size of the subset that needs to be active, the probability that your random strings will actually catch the bug decreases exponentially. For any _fixed_ target size k (the number of features that need to be active), the overall complexity is still polynomial in n (the number of existing features). But if k is a constant fraction of n, then this technique takes exponential time in n.
Imagine a scenario where your upstream dependency started out with one failure per 1,000,000 machine hours, then removed a zero once every 12 months. If you had 100 machines running tests at 100% efficiency, the bug would hit about once a year for the first year, then 10x the next year, and so on.
Put another way, if upstream is malicious, and you're not auditing every line of their source code, you're screwed.
We had a large integration test suite. It made calls to an external service, and took ~45 minutes to fully run. Since it needed an exclusive lock on an external account, it could only run a few tests at a time. We started getting random failures, so we were in a tough spot: bisecting didn't work because the failure wasn't consistent, and you couldn't run a single version of a test enough times to verify that a given version definitely did or didn't have the failure in any practical way. I ended up triggering a spread of runs over night, and then used Bayesian statistics to hone in on where the failure was introduced. I felt mighty proud about figuring that out.
Unfortunately, it turns out the tests were more likely to pass at night when the systems were under less strain, so my prior for the failure rate was off and all the math afterwards pointed to the wrong range of commits.
Ultimately, the breakage got worse and I just read through a large number of changes trying to find a likely culprit. After finally finding the change, I went to fix it only to see that the breakage had been fixed by a different team a hour or so before. It turned out to be one of our dependencies turning on a feature by slowly increasing the probability it was used. So when the feature was on it broke our tests.