News for hackers. Stuff that matters.
Re:Maybe getting rid of your QA team was bad by Zelphyr on Thursday December 14, 2023 @03:42PM (#38646961)
I worked at two companies 15-20 years ago that invested in top-tier QA teams. They were worth their weight in gold. The products were world class because the QA team were fantastic at finding bugs we developers didn't think of looking for because we were too close to the problem. We are too used to looking at the happy path.

One key attribute to both companies is that it was dictated from on high that the QA team had final say whether the release went to production or not.

These days companies think having the developers write automated tests and spend an inordinate amount of time worrying over code coverage is better. I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work.

I'm not saying automated testing is bad. I'm saying, just as the author does, that doing away with human QA testers is.

Re:Maybe getting rid of your QA team was bad by supportengineer on Thursday December 14, 2023 @03:49PM (#38647039)
I've seen QA/QE greatness and it was similar to how you describe. A different chain of command for deciding if releases are certified for production. Different incentive structures as well.

Not to mention, at one recent employer, the QE team wrote an enormous amount of code to perform their tests - It was more LOC than the modules being tested/certified.

Re:Maybe getting rid of your QA team was bad by eitally on Thursday December 14, 2023 @04:30PM (#38647518)
I had a team like that once. It was glorious. And ultimately, I'm convinced it led to overall faster development cycles because the baseline code quality & documentation was so much better than it would have been without such a great QA manager. The QA team, of course, was also technical -- mostly with SWE backgrounds -- and they were primarily colo'd in the same office as the dev team. I still remember the epiphany everyone had one planning cycle when it was mutually understood that by generally agreeing to use TDD, the QA team could participate actively in the real engineering planning and product development process.

... Then I left and my CIO let go the onshore QA team in favor of near term cost savings. Code quality went way down and within a year or two several apps needed to be entirely rewritten. Everything slowed down and people started pointing fingers, and before you knew it, it was time for "cloud native rearchitecting/reengineering" which required an SI to come in with "specialists".

Re:Maybe getting rid of your QA team was bad by MichaelZuo on Friday December 15, 2023 @06:56AM (#38653245)
I've heard this enough times that I'm becoming convinced there's a lot of 'reinventing the wheel' busywork going on in the industry.

And not even superior wheels, lower quality, more fragile wheels by clumsier wheel designers.

Re:Maybe getting rid of your QA team was bad by esafak on Thursday December 14, 2023 @04:37PM (#38647611)
So how was QA incentivized?
Re:Maybe getting rid of your QA team was bad by munificent on Thursday December 14, 2023 @06:10PM (#38648705)
> It was more LOC than the modules being tested/certified.

So much code. I hope they had a QAQA team to test all that.

Re:Maybe getting rid of your QA team was bad by fizx on Thursday December 14, 2023 @04:25PM (#38647456)
How often did you release?
Re:Maybe getting rid of your QA team was bad by SkyPuncher on Thursday December 14, 2023 @06:13PM (#38648743)
That's my question as well. Bug free code is not the goal. Valuable product is.
Re:Maybe getting rid of your QA team was bad by Zelphyr on Friday December 15, 2023 @04:01PM (#38658778)
At the first company we released a few times a year. You have to remember that CI/CD wasn't a thing back then. While I worked on a web-based product at this company, it was a time when it was still very common to release software on physical media like CD's.

At the second company we integrated Agile so we released every two or weeks (I can't recall which). The first time I heard about CI/CD was at this company from co-workers who came back from a conference all excited about it.

Re:Maybe getting rid of your QA team was bad by alkonaut on Thursday December 14, 2023 @04:27PM (#38647488)
I love my really thorough QA’s. Yes it’s an antipattern to let me as a dev lean too much on them catching what I won’t. But where I dread even running the code for a minute, they enjoy it. They take pride in figuring out edge cases far beyond any spec. They are definitely worth their weight in gold. It lets developers have confidence when changing things in the same sense a good type system does. For some classes of very interactive apps (e.g games) having unit and integration tests just doesn’t cover the parameter space.
Re:Maybe getting rid of your QA team was bad by ponector on Thursday December 14, 2023 @05:16PM (#38648093)
People here are talking about skillful QA worth their weight in gold.

Unfortunately people in the industry who have actual power in planning budgets don't think so. An article is right. QA engineers now are viewed as janitors: no one respects then, better to outsource to cheap location.

Re:Maybe getting rid of your QA team was bad by bluGill on Thursday December 14, 2023 @07:13PM (#38649362)
Depends on industry. Some care about quality, they respect qa.
Re:Maybe getting rid of your QA team was bad by PH95VuimJjqBqy on Thursday December 14, 2023 @04:30PM (#38647524)
The QA culture has to be there, not just dictates that the QA has final say.

I've seen companies where that's true and it was still trash because the QA were mostly low-paid contract workers who only did exactly what they were told and no more.

Re:Maybe getting rid of your QA team was bad by onlyrealcuzzo on Thursday December 14, 2023 @04:55PM (#38647851)
Worked at a company where QA had the final say - and that was by far the most toxic / worst environment I have ever been in.

QA also REFUSED to let developers write automation tests, also REFUSED to let us run them ourselves.

What a nightmare.

YMMV, but just having the final say is not a silver bullet for sure.

Re:Maybe getting rid of your QA team was bad by mikestew on Thursday December 14, 2023 @05:01PM (#38647919)
That’s not because “QA had the final say”, it’s because your QA team were ass clowns. Any QA team that discourages dev from writing or running tests needs to be burnt to ground and rebuilt.
Re:Maybe getting rid of your QA team was bad by ponector on Thursday December 14, 2023 @05:09PM (#38648010)
Unless dev is switching to write test 100% of time and become a tester it is highly not recommended to let developers write tests.

That QA was not a clown, they've seen some shit...

Would you let QA to write features in your production code?

Re:Maybe getting rid of your QA team was bad by NegativeK on Thursday December 14, 2023 @08:44PM (#38650102)
I spent a long time in QA.

Devs that don't test their own code are usually wasting the QA team's time with garbage. It also tends to cultivate (or is a symptom of) an environment where groups are throwing projects over the wall, so to speak, without tightly integrating QA into the process. This wastes significant amounts of time.

Re:Maybe getting rid of your QA team was bad by PH95VuimJjqBqy on Friday December 15, 2023 @08:52AM (#38654159)
that's exactly what was being described, moat building. It's political rather than effective.
Re:Maybe getting rid of your QA team was bad by ahtihn on Thursday December 14, 2023 @05:34PM (#38648307)
Test code is just code. If you can write test code you can write production code. If you can write production code you can write tests.

If your concern is that devs don't have the right mindset for testing, you can have them collaborate with a QA specialist to define the test cases and review the test implementation.

Re:Maybe getting rid of your QA team was bad by ponector on Thursday December 14, 2023 @05:50PM (#38648487)
In theory yes.

In practice, devs will not write good tests for their features, QA will be kept away from committing to production code.

Btw, if it is just code - why developers cannot implement features without bugs? No need in QA in such ideal world.

Re:Maybe getting rid of your QA team was bad by PH95VuimJjqBqy on Friday December 15, 2023 @08:52AM (#38654166)
better yet, developers write their own tests, QA write their own tests.

QA is a safety net, but they are not the first line of defense.

Re:Maybe getting rid of your QA team was bad by mdavidn on Thursday December 14, 2023 @07:21PM (#38649429)
Developers need to write tests so they understand how to structure the application in a manner that can be tested. The test engineer brings a fresh set of eyes and different expectations about how the application should work. That's valuable too.
Re:Maybe getting rid of your QA team was bad by VHRanger on Thursday December 14, 2023 @05:04PM (#38647963)
You see that with devops often. When they are incentives to block stuff instead of enabling the product they become the roadblock team.

It's why Google has SRE instead of devops people largely

Re:Maybe getting rid of your QA team was bad by mlrtime on Friday December 15, 2023 @09:15AM (#38654350)
SRE's through Error Budgets can also block stuff. They just use a different tool and metric.

Also, the SREs should be knowledgable enough to work on the fix, not just block.

Re:Maybe getting rid of your QA team was bad by Aachen on Friday December 15, 2023 @09:44AM (#38654662)
this is the internet, where saying a thing doesn't make it true.

have at your claims good sir.

Re:Maybe getting rid of your QA team was bad by ponector on Thursday December 14, 2023 @05:02PM (#38647929)
I've seen developers writing tests with no assertions, or with assertTrue(true). Always green, with 100% coverage!

The same people have been asking: why should I write tests if I can write new features?

And then one senior QA comes and destroys everything.

Once I found that if I press f5 50 times during a minute then backend will go in outOfMemory while spinning requests to the database.

Re:Maybe getting rid of your QA team was bad by feoren on Thursday December 14, 2023 @05:45PM (#38648431)
> I've seen developers writing tests with no assertions

This can be OK if the code executing without throwing exceptions is itself testing something. If you have a lot of assertions written directly into the code, as pre- or post-conditions for instance. But I'm guessing that wasn't the case here.

Re:Maybe getting rid of your QA team was bad by WrongAssumption on Thursday December 14, 2023 @08:57PM (#38650223)
Why would it be ok to not test the assertions would be triggered if the conditions are bad? How would you verify that the assertions are correct with just a happy path? If you run a code with pre-post assertions, then remove all the assertions the same test will continue to pass.
Re:Maybe getting rid of your QA team was bad by feoren on Friday December 15, 2023 @01:07PM (#38656786)
Agreed: such tests are not sufficient to test the pre- and post-conditions themselves, and are definitely happy-path focused. But such tests may be sufficient to test the happy-path code itself (which the in-code checks are also testing). It would be ridiculous to have an entire test suite with no assertions for these reasons, but it's not ridiculous (in isolation) to see some unit tests with no assertions. In theory -- and this is not advice: theory and practice are different -- but in theory, if you've absolutely air-tightly tested the "happy path", then you've basically proven no other paths occur (short of cosmic rays).

But you could actually make this same argument about unit tests with assertions: do you have any test that those assertions are written correctly? Do you unit test your unit tests? Pre- and post-conditions living "in situ" are the equivalent thing to "control" and test assertions (respectively) living in your unit tests. Would you be more comfortable if the in-situ checks were simply cut/pasted into the unit test? I wouldn't be!

Re:Maybe getting rid of your QA team was bad by kcb on Thursday December 14, 2023 @05:23PM (#38648175)
That's why the dev teams tests and the QA teams tests are not mutually exclusive.
Re:Maybe getting rid of your QA team was bad by ponector on Thursday December 14, 2023 @06:02PM (#38648612)
Testing is continuous multilayered multistaged process.

Testing should start before first developer wrote first line of code for the project. Architecture blueprints, set of requirements should be tested as early as possible. But that is not happening in real life, only in books. In real life pm will bring cheap contractor from India one month before the target release date.

Re:Maybe getting rid of your QA team was bad by mikestew on Thursday December 14, 2023 @05:03PM (#38647941)
I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work.

That’s because code coverage doesn’t find the bugs that result from code you didn’t write, but should have. Code coverage is but one measure, and to treat it as the measure is folly.

(But, yes, I have heard a test manager at a large software company we’ve all heard of declare that test team was done because 100% coverage.)

Re:Maybe getting rid of your QA team was bad by thaumasiotes on Thursday December 14, 2023 @11:54PM (#38651129)
It's more because code coverage isn't measured correctly. What you want is branch coverage.

If you have something like this:

    if condition1:
       do_something1()
    if condition2:
       do_something2()
    if condition3:
       do_something3()
There are 8 possible paths for the code to follow here, but you can cover 100% of lines of code in one test. If your code coverage measure tells you that testing the case where conditions 1, 2, and 3 are all true achieves 100% coverage, your measure is worthless. Covering this code cannot be done in less than eight tests.
Re:Maybe getting rid of your QA team was bad by alternatex on Friday December 15, 2023 @12:49AM (#38651386)
I think most code coverage analysis tools are smart enough to consider flow control statements. At least when it comes to C# ones.
Re:Maybe getting rid of your QA team was bad by fho on Friday December 15, 2023 @12:57AM (#38651427)
They do, but that's not the point GP made. An example how this could fail is if the first branch sets up something that branch two uses. Two tests are written that call branch 1&2 and not call them. 100% code coverage, even the non-happy path was tested.

But in reality, if those branches have any interaction, you would need to write eight test cases for every combination of branches being run and not being run.

Re:Maybe getting rid of your QA team was bad by thaumasiotes on Friday December 15, 2023 @06:11AM (#38653001)
Well, again, eight is a lower bound. If the software uses any values beyond the bare minimum (here) of three separate booleans, you're likely to need more.

Take this simplified code:

    if condition1:
       do_something()
You need one test for code coverage and two for branch coverage, but if there's a bug in the code as written you'll find that you actually need more. Assume that we want to run do_something() when, and only when, condition2 is true.

Now we have four cases:

- condition1 && condition2: do_something() runs, which is correct.

- condition1 && !condition2: do_something() runs, which is incorrect.

- !condition1 && !condition2: do_something() does not run, which is correct.

- !condition1 && condition2: do_something() does not run, which is incorrect.

If you happened to write tests for the first and third of those options, your tests will make it look like your code works, and your (official) coverage metrics will look comprehensive. But because your coverage metrics (in reality) are actually terrible, your code is secretly not working.

The big problem we've just run into is that the number of test cases we need depends on the number of bugs in our code. If the goal of writing tests is to prove that we don't have bugs, this is a terrible result; it means our goal is fundamentally impossible to achieve. We don't know whether we have bugs, so we don't know how many tests we need.

Re:Maybe getting rid of your QA team was bad by thaumasiotes on Friday December 15, 2023 @01:54AM (#38651710)
Why do you think that? It isn't true.
Re:Maybe getting rid of your QA team was bad by mikestew on Friday December 15, 2023 @01:35PM (#38657066)
No, it's more because when the bug was filed, you said to yourself, "oh, was I supposed to update the $HOOZIWIZIT in that transaction, too? Oops." Code coverage won't save you from that. Matching test cases to requirements, or summat, can be helpful, but good golly it's a bunch more paperwork.

Your point is a good one, but not where I was going.

Re:Maybe getting rid of your QA team was bad by bluGill on Thursday December 14, 2023 @07:11PM (#38649345)
Automated tests take manual testing from 60% to 50% of the time to develop quality software. Valuable, but not a bullet to manual tests.
Re:Maybe getting rid of your QA team was bad by hedora on Thursday December 14, 2023 @09:41PM (#38650547)
The best approach I’ve seen is to do all of the above.

- Have QA run pessimal versions of real use cases. Trying to sell a word processor to lawyers? Format the entire US legal code and a bajillion contracts in it, then duplicate it 10x and start filing usability/performance bugs.

- Have the engineers test everything with randomly generated workloads before committing. Run those tests nightly, and fix all the crashes / failures.

- Have Product Management (remember them?) work with marketing and sales to figure out what absolutely has to ship, and when.

Make sure it only takes one of the above three groups to stop ship, and also to stop non-essential development tasks.

Re:Maybe getting rid of your QA team was bad by Arainach on Friday December 15, 2023 @03:56AM (#38652332)
The devil is in the details of "essential development tasks".

An uncountable number of products have died or devolved because "we don't have time to do it that way, put in the quick fix"

Re:Maybe getting rid of your QA team was bad by turok on Friday December 15, 2023 @06:25AM (#38653072)
We try to do this, QA owns deployment to production and has the final say if a feature needs a re-write. The almost adversarial incentives of dev and QA are why it works. Dev wants to close ticket, QA wants the feature to work, Product wants to tick a box, all collaborate so that the closed ticket delivers a working feature and not just one of those three things.