News for hackers. Stuff that matters.
Re:Questions to ask yourself when writing tests by crdoconnor on Monday January 01, 2018 @09:27PM (#16049400)
>Can I do this faster if I don’t write a test? The answer is always no.

I think the answer is mostly no but it's dangerous to think that it's always no.

I've been given many stories in the past for which writing a realistic automated test would have taken days, manual verification took minutes and the code was fairly isolated and did not get changed very frequently.

Writing a test under those circumstances is actually a pretty poor investment.

Re:Questions to ask yourself when writing tests by simonhamp on Monday January 01, 2018 @10:15PM (#16049590)
I would be very surprised if an automated test literally took days to write when a manual test is just minutes.

Also the time savings still pay off later, as automated tests usually take seconds to run and there’s no training required - once it’s in the test suite and the test suite runs are automated, it will always run and quickly identify a failure - no “oops, we forgot to show Jim the Intern that he had to test that part manually...”

Setting up your test suite and automation is longer for sure, but not days. Even a complex manual process can be automated relatively quickly... the manual process should be fairly scriptable in any OS nowadays, and most platforms have great test frameworks.

Re:Questions to ask yourself when writing tests by crdoconnor on Tuesday January 02, 2018 @03:49AM (#16050739)
Trust me you'll come across examples like this one day if you make a concerted effort to automate a test for every story. It's a rough guess but I'd say I don't automate 1 out of 20 stories - I'm not exactly blithely unaware of the benefits an automated test suite brings.

Most of the examples that require "days" (or weeks) for the test and 5 minutes for the test would involve the building or amending an elaborate mock/stub.

Some examples where this happens include rare interactions with weird hardware, odd legacy APIs that are scheduled to be replaced, race conditions and obscure edge cases with mocked services.

I document all of these special cases and they should clearly remain special but I'm not going to blindly assume that automating the story will have a positive ROI.

Re:Questions to ask yourself when writing tests by andrewflnr on Monday January 01, 2018 @11:56PM (#16049966)
Think about something that necessarily involves hardware interaction, or a GUI, or where all the interesting error cases are non-deterministic (concurrency, network error handling). Ok, on the last one you're pretty much hosed anyway. But we're not all writing nice data-in/data-out apps.
Re:Questions to ask yourself when writing tests by crdoconnor on Tuesday January 02, 2018 @09:10AM (#16051887)
Actually, I probably wouldn't count network error handling, because you can probably use something like vaurien - https://github.com/community-libs/vaurien - to deterministically mimic bad network conditions in an integration test in a fairly reasonable amount of time.

Also, integrating that tool would probably have applications beyond a single story so even if the change takes 5 minutes and making the test work with vaurien takes half a day, it's probably still worthwhile.

Assuming that no tool like vaurien exists, though (and there are plenty of scenarios out there where you'd have to build it from scratch), building the test scenario could become prohibitively expensive.

Re:Questions to ask yourself when writing tests by falsedan on Tuesday January 02, 2018 @05:46AM (#16051137)
Let me help expand your imagination: what if the code is for sshing to prod and needs a ssh-agent or a password typed in? Would you leave those cress on your CI system? Would Security be happy with that?
Re:Questions to ask yourself when writing tests by albertgoeswoof on Tuesday January 02, 2018 @07:26AM (#16051472)
Why would a test build ever ssh to production?

If you absolutely need to test connecting to an external system you should mock it or create an environment for testing against.

Your credentials should also be stored as env variables so the test ones are used only for the test environment.

Re:Questions to ask yourself when writing tests by falsedan on Tuesday January 02, 2018 @01:20PM (#16053923)
Because you're developing automated tools to deploy to prod & you're verifying the correctness of the system end-to-end with a no-op command. The connecting to an external system is not what's under test, and assume I've written unit tests that mock out the sshing, and that even with a staging environment I have to smoke test the new code before switching everyone over to the new version.

Where are those env variables stored on the CI system? In the job config? I would use Vault for this.

Re:Questions to ask yourself when writing tests by KirinDave on Monday January 01, 2018 @10:07PM (#16049557)
That's a decision you can make, but all you're doing is costing yourself more time later for the benefit of less time now.
Re:Questions to ask yourself when writing tests by crdoconnor on Tuesday January 02, 2018 @03:40AM (#16050704)
5x 5 minute manual test < 1 day

Automatd test fundamentalists are subject to the same kinds of folly that other fundamentalists are.

Re:Questions to ask yourself when writing tests by KirinDave on Tuesday January 02, 2018 @12:13PM (#16053293)
And when someone else modifies the code and introduces a subtle bug, will you attribute the costs handling that to this same timebucket?

How many customer minutes lost = 1 developer minute?

That said, I think tests are the most expensive and most brittle way to address this problem. They're necessary, but should be deployed sparingly.

Re:Questions to ask yourself when writing tests by crdoconnor on Tuesday January 02, 2018 @01:19PM (#16053907)
Yes, I would. However:

A) In such cases I think testing against the real thing often has a greater chance of catching subtle bugs rather than the automated test scenario against the elaborate mock which is highly likely to share many of the same assumptions that caused the bug.

B) Code reviews ought to flag that a piece of code that is not under automated test is being modified and appropriate care should be taken (ideally this alert should be automated but I haven't yet reached that level).

I think it pays to approach these things on a case by case basis, and if a pattern of subtle bugs does appear that's a strong indication that you should change your behavior (I'm a stronger believer in bug retros than I am in any kind of testing dogma).

>How many customer minutes lost = 1 developer minute

Is that a relevant question to ask? If you introduce the presumption that an automated scenario test is more likely to catch a bug than a manual test then I guess it's relevant, but honestly for these types of scenario I think the opposite is true.

I didn't mention it before also but if you have manual testers on hand that changes the calculus too. I'd say it's normal for 3-4 manual tester minutes to be equivalent to about 1 developer minute.

As I mentioned above, I really don't think it pays to be a test fundamentalist.

>That said, I think tests are the most expensive and most brittle way to address this problem.

Or a type system fundamentalist.

Re:Questions to ask yourself when writing tests by KirinDave on Tuesday January 02, 2018 @05:04PM (#16056208)
Actually, my personal belief is that people, humans like you and I, are all awful at writing software. We're even worse at enumerating and writing tests.

Miserably bad. Unforgivably bad.

Slowly refining and testing systematizations of correct software building processes is perhaps the most important thing we can do in the first 2 centuries of "software" as a thing.

Because otherwise, all we'll do is continue to wallow in pride and failure, claiming it can't be helped. All the while using language like "case by case," that I have taken to mean: "I will never do that unless you force me to."

Fortunately, I think the scope of failure and fraud in the software industry has grown so late that folks are starting to take correctness as a requirement and not a nice to have. Another Equifax or two and maybe a nice DAO hack or something and folks are going to start saying, "Maybe it's just too bad we all learn to make bad software," turning to new techniques and practices.

Far-fetched? Maybe. But it is happening with AI...

Re:Questions to ask yourself when writing tests by crdoconnor on Tuesday January 02, 2018 @05:58PM (#16056677)
>that I have taken to mean: "I will never do that unless you force me to."

I actually created my own open source BDD framework and a ton of tooling to help automate stories.

19 out of 20 was a pretty conservative estimate of how much I automate - it's probably more like 39 out of 40. I'm a little obsessive about it because I want to dogfood my work properly so I automate quite a few things where the cost/benefit for a normal programmer would seem a bit low.

I'm very cognizant that the industry as a whole is terrible at testing and I'm hoping my software can one day do a small part to help with that.