Re:GitHub Copilot available for JetBrains and Neovim
by Graffur on Wednesday October 27, 2021 @04:08PM
(#29018459)
So one argument against code coverage requirements is that poor engineers won't test correctly. Without the code coverage requirements you're in the same situation.
Re:GitHub Copilot available for JetBrains and Neovim
by crysin on Wednesday October 27, 2021 @04:53PM
(#29018970)
Problem is with 100% code coverage of badly guarded / implemented code you'll have a fall sense of security if you're just looking at coverage as the metric of quality. Anytime I've worked with a company who had a required code coverage percent, they never actually cared what the code being covered looked like only that it was covered in some test.
Re:GitHub Copilot available for JetBrains and Neovim
by manquer on Wednesday October 27, 2021 @07:21PM
(#29020308)
Without going down the rabbit hole of Goodhart's law, code coverage % is a poor metric particularly when used standalone.
Re:GitHub Copilot available for JetBrains and Neovim
by jmnicolas on Wednesday October 27, 2021 @04:19PM
(#29018592)
I don't think there's Infinity in my language, what do you use it for except maths?
Re:GitHub Copilot available for JetBrains and Neovim
by dpryden on Wednesday October 27, 2021 @05:54PM
(#29019594)
If you are using floating point numbers implemented in hardware, then infinity is absolutely a valid value and one that your code will encounter. This is true regardless of language, as long as the language requires or allows IEEE-754 semantics.
I am not aware of any language (outside of intentionally-minimalist esolangs) that doesn't support floating point numbers. In some languages (like JavaScript) that's the only kind of number you get.
I have seen projects where a test will have sequence of steps which trigger the code but the assertion is effectively true === true, or they will replicate the same function in the test instead of generating proper mock data or myriad different absurd testing approaches. This comes from twin pressure of showing coverage and having tests pass.
Coverage is also a challenge in code which uses AI/ML libraries or that use third party services. These really need statistical testing with large volume of diverse well maintained samples and results need statistical analysis for error rates not different from how manufacturing does it, I don't see that often. For code using face detection for example a single face getting detected or not is hardly an adequate test.
Finally, it is easier to improve coverage by testing simpler code than improve coverage/ refactor a function which has say 10 nested branches, so it is not uncommon to see 90% coverage and 10% of the most used / most error prone code poorly or not tested at all.
There are some methods to address these like mutation testing, do retros for failure of tests to capture production bugs, it is not easy to measure and coverage driven orgs will not see their metrics moving by doing these.
Well written test suites will also have good coverage, but not necessarily other way around. Developers who care and understand what they are doing and why they are doing it, will use coverage as only the first step to see where there are gaps in their tests.
Tests are also code that need peer reviewed and maintained, if the tests depend on implementation and constantly break or contains improper mocks or assert poorly. A lot of not well written tests is hindrance to development than aid it.
[1] Yes, most of these are not applicable in a strongly typed language, but it is far easier as a illustration .