News for hackers. Stuff that matters.
Re:Superior Testing: Stop Stopping by kstenerud on Thursday January 31, 2019 @04:45AM (#19043637)
Regardless of all the arguments for and against tests, it's important to remember your purpose as an IT professional: Your job is to solve the customer's problem. That's it.

HOW you solve their problems is entirely up to you, and some solutions will work better than others.

The only thing about customers guaranteed to be consistent is that they will request changes. Some will be good changes. Some will be bad changes. Some will be unavoidable changes. Your job here is to guide them through the best possible change process, with the ultimate goal of solving their problems.

Testing is a useful tool for (1) making sure the system matches the customer's expectations, and (2) making sure new changes don't break old things.

How you implement tests is entirely up to you, but once again, some solutions work better than others depending on the situation. Manual testing only scales up to a point before the cost of the manual testing is greater than the cost of implementing and maintaining automated tests. You need to become good at estimation here, but for 90% of projects, automated testing is the most efficient long term strategy.

WHERE you test is also important. Generally, it's best to test components at the edge of their interfaces. When you input X, Y should come out, ideally every time. This is where immutable data and idempotent APIs are VERY helpful for maintaining a reliable codebase. The more you need to cut into a component's innards to test it, the more you should be asking yourself why.

Testing is very much about architecture. Make clear boundaries between components and outside interfaces, use immutable data and idempotent APIs, and tests become a lot easier to write, and more resilient to change.

Re:Superior Testing: Stop Stopping by olau on Thursday January 31, 2019 @08:07AM (#19044407)
> but for 90% of projects, automated testing is the most efficient long term strategy

I don't think this is a good way to think about it.

I see plenty of projects that don't go anywhere, and I also see plenty of projects that do go somewhere, but where large parts change so little and are so easy to test manually that there's little point in much automation. Other parts are complex and full of contradictory business rules so formal tests are necessary.

Of course, YMMV. If you're inside Google, things are probably looking very different. But most people aren't.

Re:Superior Testing: Stop Stopping by bunderbunder on Thursday January 31, 2019 @10:32AM (#19045427)
I've inherited one or two projects like the first one you describe before, and, when they don't have tests, it's awful.

The problem is, regardless of how stable the code was, or how easy it was for the original author to test it manually, if the code hasn't been worked on for ages, then the original author is either gone or can't remember all the details of how it was supposed to work anymore. Which leaves you in a dangerous situation, because there's no good way of knowing which behaviors are by design and which ones are incidental.

Re:Superior Testing: Stop Stopping by drdeadringer on Thursday January 31, 2019 @03:25PM (#19048451)
Documentation.

I swear, if it's not "we don't have time to test" it's "we don't have time to document". Ok, but you've hired all these firefighters instead.

Re:Superior Testing: Stop Stopping by bunderbunder on Thursday January 31, 2019 @03:30PM (#19048499)
Re:Superior Testing: Stop Stopping by TeMPOraL on Friday February 01, 2019 @03:12AM (#19052490)
Also, if you've inherited code, or didn't work on it for couple weeks, chances are you won't know/remember how to do the manual tests, and will waste time trying to rederive it from first principles. This also applies to any other kind of "ad-hoc" ops, and multiple times more so if the activity involves multiple runtimes - e.g. issuing a shell command. I've learned this the hard way, and these days I note down the precise steps/commands for any manual op that's not very clearly a one-time thing / throwaway. Your future self and whoever inherits your code will both thank you.
Re:Superior Testing: Stop Stopping by gnahckire on Thursday January 31, 2019 @01:10PM (#19047110)
Adding in to your point, it's really easy to get familiar with a codebase and be confident that there are no regressions when you make changes if there are comprehensive tests.

Similarly, tests can serve as a device for asserting contributions are acceptable pre/post-merge as well as evaluating that your dependencies work as expected.

Re:Superior Testing: Stop Stopping by millstone on Thursday January 31, 2019 @06:20AM (#19043995)
There's a case for ugly entangled tests too. If you, say, isolate your tests from the filesystem (mocks, etc), your tests can become fast, hermetic, etc. But they lose all sensitivity to differences between filesystems, and encode any assumptions about how filesystems behave.

I like to have tests that exercise the real stuff. The tests are flakier but have better fidelity to the actual use.

Re:Superior Testing: Stop Stopping by skybrian on Thursday January 31, 2019 @01:14PM (#19047152)
I generally agree, but we should acknowledge that "the real stuff" can be, in its way, somewhat fake.

Portable code can be run on whatever filesystem the user chooses. We can only test with what we have. So this boils down to another way to say "works on my machine" and that may or may not be relevant depending on how well the OS and hardware we test on matches the customer's setup.

It's good to be able to run your tests against whatever filesystem you choose, though.

Re:Superior Testing: Stop Stopping by bunderbunder on Thursday January 31, 2019 @02:23PM (#19047842)
Concrete example: Docker for Linux, Docker for Mac and Docker for Windows all implement both networking and mounted volumes in slightly different ways.

This has recently eaten up a nonzero amount of my time in maintaining our integration and acceptance test suites.

I'm not sure where I'm going with this, other than to observe that real-ish isn't necessarily a substitute for real. I suppose the "developer empathy" story is that maybe the more empathetic approach in the long run would have been to have developers work on the target platform rather than letting them pick whatever one they used to use at their old gig.