News for hackers. Stuff that matters.
Re:All of us test in production all the time (2019) by siliconc0w on Sunday July 19, 2020 @04:41PM (#23892273)
So it's a fair point that the fidelity of non-prod environments are inherently limited and you still need a bunch of other stuff like canaries, automated canary analysis, automated rollbacks, zone fault-tolerance, feature-flags, chaos engineering, server and client-side instrumentation, but this is generally 'next-level' stuff when most shops aren't even getting the basics right.

For the overwhelming majority of non-prod environments and release processes I've seen they usually have basic problems and would catch a lot more issues with additional engineering investment. I.e Releases aren't automated, self-contained, or idempotent. Unit testing coverage is bad, integration tests are non-existent, data used isn't reflective of reality, there is no performance or load testing done, downstream or upstream systems have poorly defined interfaces and are excessively mocked or just not considered.

Re:All of us test in production all the time (2019) by faizshah on Sunday July 19, 2020 @08:01PM (#23893461)
Wholeheartedly agree to everything you’ve said.

I have worked at a lot of places where automated testing is almost non existent. Instead they use staging/testing environments and manual testing.

For all these projects they frequently have downtime or errors in prod where something was uncaught. It takes ages to release a fix because they have to carefully research how a change affects other code instead of running a test suite.

They are convinced that unit tests are unnecessary extra work for their project but they don’t realize they are losing time by having to manually test and research changes. I think a lot of the focus on fancy testing practices has lead some programmers to think testing is a load of extra work when things will fail in prod anyway. We need to push back and say automated tests are a way to codify assumptions about the code and help speed up the prototype-build-test-release loop.

Side-note: self documenting code is a myth but I digress.

Re:All of us test in production all the time (2019) by anon9001 on Sunday July 19, 2020 @08:59PM (#23893726)
> They are convinced that unit tests are unnecessary extra work for their project but they don’t realize they are losing time by having to manually test and research changes.

If people are finding tests unnecessary extra work, maybe they're right for their project.

Most jobs are just "ship feature asap, try not to break anything, repeat". Most managers don't care about code quality as much as speed of delivery. Most coders don't have much "skin in the game" and can always get another job when the project fails.

Writing tests is a sign that you know what you're building, you have a methodology for building it, and you have a commitment to keeping it working in the future. That's simply not the requirements for most programming work, and it can be a disservice to both the coders and the management to push for testing that won't help achieve the business goals. If a business needs to iterate fast and reliably, write tests. If a business needs to iterate fast and is ok with breaking things, skip the tests and have a rollback plan instead.

Re:All of us test in production all the time (2019) by faizshah on Sunday July 19, 2020 @09:34PM (#23893910)
Sorry, I have to completely disagree with this. This is a horrible, but often used, excuse.

In my experience these coders are doing exactly what you would do in a unit test but instead they are doing it manually by adding printlns and testing various values on the command line, in a repl, a notebook, or a main. Instead of manually adding these things every time you want to test your work, if you just added some basic unit tests it would speed up your delivery. Not only that but you can't "forget to test that" because you're forced to write down the test cases instead of deleting them when you're done.

If you add up all the time you spent adding printlns to your code, testing various input values in a main method, researching inside your codebase to see what could fail, debugging your ETL because you forgot about an uncaught error or a different kind of null in this column that only came up after it processed 3TB of data (real example), etc. I guarantee you would save time if you just implement some basic automated tests instead of testing in prod or staging.

Re:All of us test in production all the time (2019) by cs02rm0 on Monday July 20, 2020 @01:35AM (#23894932)
I don't disagree, probably.

I keep finding myself working on internal projects that are driven by a UI and have little business logic but lots of I/O to other systems. With no hard requirements, no guarantee of shipping and no direct access to end users. Which I find less suited to unit tests than (automated) integration testing. As soon as you put unit testing in all anyone can ask is what your test coverage percentage is and why it's some value less than 100.

Unit testing is a dev tool, if a dev sees value in writing and maintaining a given test, great, otherwise it could just be that it's not a panacea. Let them strike a balance but ensure the team ask the question of any production issues - could a unit test have prevented this?

Re:All of us test in production all the time (2019) by hampfi on Monday July 20, 2020 @07:22AM (#23896720)
Yeah no.

The biggest issue in all my dev work so far was a really shitty codebase which would work somehow and had and still have hidden big issues going on.

You just don't see it today perhaps and delivered it in 1 day instead of 3. But you will fix the issue in production half a year later and either have lost data, corrupted data or you have to correct data.

Not doing any tests, is literally shit.

Missing Unit tests also often means no proper CI/CD pipeline. Because yeah why would you need a proper build if you don't even execute tests anyway right?

IT IS always a clearly ignored requirement of doing proper work. Quality work. Do less, do better. Its the fault of the manager to not think about it and enforce it and its the fault of the developer to build shit.

Re:All of us test in production all the time (2019) by ritchiea on Monday July 20, 2020 @01:31AM (#23894911)
Managers need developers to write tests so that when that developer job hops the new dev on the team can actually make changes with confidence. As it happens, writing tests and adhering to best practices has a side effect of keeping turnover low because developers can do their jobs with less friction.
Re:All of us test in production all the time (2019) by t-writescode on Monday July 20, 2020 @01:24AM (#23894885)
Then, may I present the argument that, in general, you should be testing if for one reason: to get practice writing tests for when you actually have a project that you need to write tests in (edit: according to your metric).
Re:All of us test in production all the time (2019) by Olreich on Monday July 20, 2020 @07:17AM (#23896683)
Self-documenting code is all code. The only thing that speaks to what a piece of code does is the code. Comments aren’t syntax, so they don’t help you there. I’ve never seen a codebase that does a good job of documenting the system architecture, expected usage patterns, and why the code is built the way it is. And I don’t know that having all that be inline to the code would be helpful anyway.

I find it far less error prone to just read the code and determine its purpose and allowable modifications from that. I am thinking about what a citation or footnote system might look like for code now though. That has the potential to provide context on-demand without needing to separate out all the pieces of the code into little silos that are hard to piece together.

Re:All of us test in production all the time (2019) by faizshah on Monday July 20, 2020 @09:00AM (#23897345)
Self-documenting code is a total myth. Instead of the author documenting what a piece of code does, why it does it that way and how to use it everyone who works on that code has to spend hours looking through the commit history, merge requests, design docs, wiki history, asking around etc piecing together the history and conversations around the code.

Sometimes these artifacts don’t exist so then you have to go do the same research the author did to write the code just to figure out why the code exists...

Note that none of those activities are reading the code itself because you can do that in both scenarios.

Re:All of us test in production all the time (2019) by smoe on Sunday July 19, 2020 @06:36PM (#23893025)
Agreed

I like the view i picked up somewhere but don't remember from whom: Tests are not there to prove something is working, but to prove something is not working.

They are a fairly cheap way to tell me during development that this ain't going to work no matter what. And if everything compiles and the tests and reviews pass, I think this has a good shot at working out in production, but there still might well be cases and circumstances I haven't or coulnd't have considered.

No setup will cover 100%, but with not that much work you can already get quite far. The tricky thing in my opinions is more on deciding which kind of tests, processes and automations are actually worth implementing and maintaining at a given stage of a project.

Re:All of us test in production all the time (2019) by table-delete on Sunday July 19, 2020 @07:46PM (#23893372)
Glenford Myers in "The Art of Software Testing" said: "Testing is the process of executing a program with the intent of finding errors."

The preface to this is even more enlightening:

"When you test a program, you want to add some value to it. Adding value through testing means raising the quality or reliability of the program. Raising the reliability of the program means finding and removing errors.

Therefore, don’t test a program to show that it works; rather, you should start with the assumption that the program contains errors (a valid assumption for almost any program) and then test the program to find as many of the errors as possible."

Re:All of us test in production all the time (2019) by faizshah on Sunday July 19, 2020 @08:05PM (#23893484)
I like this quote from Kent Beck:

“ It is impossible to test absolutely everything, without the tests being as complicated and error-prone as the code. It is suicide to test nothing (in this sense of isolated, automatic tests). So, of all the things you can imagine testing, what should you test?

You should test things that might break. If code is so simple that it can't possibly break, and you measure that the code in question doesn't actually break in practice, then you shouldn't write a test for it...

Testing is a bet. The bet pays off when your expectations are violated [when a test that you expect to pass fails, or when a test that you expect to fail passes]... So, if you could, you would only write those tests that pay off. Since you can't know which tests would pay off (if you did, then you would already know and you wouldn't be learning anything), you write tests that might pay off. As you test, you reflect on which kinds of tests tend to pay off and which don't, and you write more of the ones that do pay off, and fewer of the ones that don't.”

Source: https://softwareengineering.stackexchange.com/a/244709

Re:All of us test in production all the time (2019) by josephg on Sunday July 19, 2020 @10:20PM (#23894134)
When I’ve gone full TDD on projects, I’m always surprised to find there’s a power law distribution or something on failing tests. Most of the tests I write never catch a bug in the life of the software. Something like 90% of the value of a test suite could be achieved with only about 10% of the tests. Of course, the trick is figuring out which tests are going to repeatedly fail ahead of time. But there’s something interesting down this rabbit hole.

I wonder if it would be worth collecting per-test stats through the life of a project to explore this.

Re:All of us test in production all the time (2019) by lmm on Sunday July 19, 2020 @10:42PM (#23894210)
You'd get a lot of the value by just being willing to delete tests once it's clear they're not serving any purpose. Sadly a lot of people would rather see a high test coverage percentage than have an effective test suite.
Re:All of us test in production all the time (2019) by nitrogen on Monday July 20, 2020 @10:18PM (#23903785)
What is the value of deleting a test once it's written, though?
Re:All of us test in production all the time (2019) by lmm on Monday July 20, 2020 @10:30PM (#23903840)
Tests impose a maintenance burden like any other lines of code. Just making it harder to navigate to relevant code (including useful tests) is a significant cost.
Re:All of us test in production all the time (2019) by ornornor on Tuesday July 21, 2020 @09:01AM (#23906467)
You’ve already paid the cost for writing the test, might as well keep it.

If a test fails when modifying the codebase or adding a feature or updating dependencies, it still serves a purpose in my opinion even if fixing it takes some time.

Having the test no matter how trivial also helps boost confidence that a new change didn’t break anything no matter how trivial.

But tests should be refactored once in a while, and if a thing is complicated to setup for testing, either the thing to test should be refactored before shipping and/or the developer working on that thing should also ship a factory and helper functions for testing that thing.

Re:All of us test in production all the time (2019) by lmm on Tuesday July 21, 2020 @10:40PM (#23913403)
> If a test fails when modifying the codebase or adding a feature or updating dependencies, it still serves a purpose in my opinion even if fixing it takes some time.

> Having the test no matter how trivial also helps boost confidence that a new change didn’t break anything no matter how trivial.

If it catches an error in your change, that's a benefit. If it delays a correct change with false positives, that's a cost. (And if a passing test "boosts confidence" but the change is actually broken, that's another, more subtle cost). Often the cost is larger than the benefit.

> But tests should be refactored once in a while, and if a thing is complicated to setup for testing, either the thing to test should be refactored before shipping and/or the developer working on that thing should also ship a factory and helper functions for testing that thing.

That's all valid assuming the tests are providing some value. But just like with any code, the first question should be whether it's serving a useful purpose at all - if not, then deleting is cheaper and more effective than any amount of careful refactoring.