Not to mention, at one recent employer, the QE team wrote an enormous amount of code to perform their tests - It was more LOC than the modules being tested/certified.
... Then I left and my CIO let go the onshore QA team in favor of near term cost savings. Code quality went way down and within a year or two several apps needed to be entirely rewritten. Everything slowed down and people started pointing fingers, and before you knew it, it was time for "cloud native rearchitecting/reengineering" which required an SI to come in with "specialists".
And not even superior wheels, lower quality, more fragile wheels by clumsier wheel designers.
So much code. I hope they had a QAQA team to test all that.
At the second company we integrated Agile so we released every two or weeks (I can't recall which). The first time I heard about CI/CD was at this company from co-workers who came back from a conference all excited about it.
Unfortunately people in the industry who have actual power in planning budgets don't think so. An article is right. QA engineers now are viewed as janitors: no one respects then, better to outsource to cheap location.
I've seen companies where that's true and it was still trash because the QA were mostly low-paid contract workers who only did exactly what they were told and no more.
QA also REFUSED to let developers write automation tests, also REFUSED to let us run them ourselves.
What a nightmare.
YMMV, but just having the final say is not a silver bullet for sure.
That QA was not a clown, they've seen some shit...
Would you let QA to write features in your production code?
Devs that don't test their own code are usually wasting the QA team's time with garbage. It also tends to cultivate (or is a symptom of) an environment where groups are throwing projects over the wall, so to speak, without tightly integrating QA into the process. This wastes significant amounts of time.
If your concern is that devs don't have the right mindset for testing, you can have them collaborate with a QA specialist to define the test cases and review the test implementation.
In practice, devs will not write good tests for their features, QA will be kept away from committing to production code.
Btw, if it is just code - why developers cannot implement features without bugs? No need in QA in such ideal world.
QA is a safety net, but they are not the first line of defense.
It's why Google has SRE instead of devops people largely
Also, the SREs should be knowledgable enough to work on the fix, not just block.
have at your claims good sir.
The same people have been asking: why should I write tests if I can write new features?
And then one senior QA comes and destroys everything.
Once I found that if I press f5 50 times during a minute then backend will go in outOfMemory while spinning requests to the database.
This can be OK if the code executing without throwing exceptions is itself testing something. If you have a lot of assertions written directly into the code, as pre- or post-conditions for instance. But I'm guessing that wasn't the case here.
But you could actually make this same argument about unit tests with assertions: do you have any test that those assertions are written correctly? Do you unit test your unit tests? Pre- and post-conditions living "in situ" are the equivalent thing to "control" and test assertions (respectively) living in your unit tests. Would you be more comfortable if the in-situ checks were simply cut/pasted into the unit test? I wouldn't be!
Testing should start before first developer wrote first line of code for the project. Architecture blueprints, set of requirements should be tested as early as possible. But that is not happening in real life, only in books. In real life pm will bring cheap contractor from India one month before the target release date.
That’s because code coverage doesn’t find the bugs that result from code you didn’t write, but should have. Code coverage is but one measure, and to treat it as the measure is folly.
(But, yes, I have heard a test manager at a large software company we’ve all heard of declare that test team was done because 100% coverage.)
If you have something like this:
if condition1:
do_something1()
if condition2:
do_something2()
if condition3:
do_something3()
There are 8 possible paths for the code to follow here, but you can cover 100% of lines of code in one test. If your code coverage measure tells you that testing the case where conditions 1, 2, and 3 are all true achieves 100% coverage, your measure is worthless. Covering this code cannot be done in less than eight tests.But in reality, if those branches have any interaction, you would need to write eight test cases for every combination of branches being run and not being run.
Take this simplified code:
if condition1:
do_something()
You need one test for code coverage and two for branch coverage, but if there's a bug in the code as written you'll find that you actually need more. Assume that we want to run do_something() when, and only when, condition2 is true.Now we have four cases:
- condition1 && condition2: do_something() runs, which is correct.
- condition1 && !condition2: do_something() runs, which is incorrect.
- !condition1 && !condition2: do_something() does not run, which is correct.
- !condition1 && condition2: do_something() does not run, which is incorrect.
If you happened to write tests for the first and third of those options, your tests will make it look like your code works, and your (official) coverage metrics will look comprehensive. But because your coverage metrics (in reality) are actually terrible, your code is secretly not working.
The big problem we've just run into is that the number of test cases we need depends on the number of bugs in our code. If the goal of writing tests is to prove that we don't have bugs, this is a terrible result; it means our goal is fundamentally impossible to achieve. We don't know whether we have bugs, so we don't know how many tests we need.
Your point is a good one, but not where I was going.
- Have QA run pessimal versions of real use cases. Trying to sell a word processor to lawyers? Format the entire US legal code and a bajillion contracts in it, then duplicate it 10x and start filing usability/performance bugs.
- Have the engineers test everything with randomly generated workloads before committing. Run those tests nightly, and fix all the crashes / failures.
- Have Product Management (remember them?) work with marketing and sales to figure out what absolutely has to ship, and when.
Make sure it only takes one of the above three groups to stop ship, and also to stop non-essential development tasks.
An uncountable number of products have died or devolved because "we don't have time to do it that way, put in the quick fix"
One key attribute to both companies is that it was dictated from on high that the QA team had final say whether the release went to production or not.
These days companies think having the developers write automated tests and spend an inordinate amount of time worrying over code coverage is better. I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work.
I'm not saying automated testing is bad. I'm saying, just as the author does, that doing away with human QA testers is.