News for hackers. Stuff that matters.
Re:Questions to ask yourself when writing tests by simonhamp on Monday January 01, 2018 @08:28PM (#16049150)
The most important question to ask when writing tests: Can I do this faster if I don’t write a test?

The answer is always no. Even if you are the only person building something, future you will lick your boots clean in gratitude if there are tests.

Because even the best developers have to work with their own code sometimes.

Re:Questions to ask yourself when writing tests by mooreds on Monday January 01, 2018 @09:08PM (#16049328)
This!

I became a believer in automated testing when I worked with a large ETL process that worked with real estate data. This data was input by real estate agents, varied wildly in quality, and had both image files and structured text data. Releasing this code before automated testing was fearful. We'd push changes to staging and then wait for three (!) days of data, then examine the staging and production databases manually and see if there were material differences.

Needless to say, we didn't like to release this code very often.

When I left, we weren't doing automated testing on the image processing, but we were on the text side of things. It became far easier to do a release because we had a set of regression tests that gave us confidence we weren't breaking anything. If something did pop up, we could add it to the suite.

Human beings systematically undervalue their future selves. This is why we have a hard time saving for retirement, exercising and writing tests. Think of your future self and write tests.

Re:Questions to ask yourself when writing tests by maksimum on Tuesday January 02, 2018 @12:12AM (#16050019)
> Human beings systematically undervalue their future selves. This is why we have a hard time saving for retirement, exercising and writing tests. Think of your future self and write tests.

This!

What's also worrisome is that sometimes your incentives are misaligned; e.g. when product development and maintenance are handled by different teams, or features have tight deadlines and screw the debt that will be the future and we must deal with now.

Re:Questions to ask yourself when writing tests by crdoconnor on Monday January 01, 2018 @09:27PM (#16049400)
>Can I do this faster if I don’t write a test? The answer is always no.

I think the answer is mostly no but it's dangerous to think that it's always no.

I've been given many stories in the past for which writing a realistic automated test would have taken days, manual verification took minutes and the code was fairly isolated and did not get changed very frequently.

Writing a test under those circumstances is actually a pretty poor investment.

Re:Questions to ask yourself when writing tests by simonhamp on Monday January 01, 2018 @10:15PM (#16049590)
I would be very surprised if an automated test literally took days to write when a manual test is just minutes.

Also the time savings still pay off later, as automated tests usually take seconds to run and there’s no training required - once it’s in the test suite and the test suite runs are automated, it will always run and quickly identify a failure - no “oops, we forgot to show Jim the Intern that he had to test that part manually...”

Setting up your test suite and automation is longer for sure, but not days. Even a complex manual process can be automated relatively quickly... the manual process should be fairly scriptable in any OS nowadays, and most platforms have great test frameworks.

Re:Questions to ask yourself when writing tests by crdoconnor on Tuesday January 02, 2018 @03:49AM (#16050739)
Trust me you'll come across examples like this one day if you make a concerted effort to automate a test for every story. It's a rough guess but I'd say I don't automate 1 out of 20 stories - I'm not exactly blithely unaware of the benefits an automated test suite brings.

Most of the examples that require "days" (or weeks) for the test and 5 minutes for the test would involve the building or amending an elaborate mock/stub.

Some examples where this happens include rare interactions with weird hardware, odd legacy APIs that are scheduled to be replaced, race conditions and obscure edge cases with mocked services.

I document all of these special cases and they should clearly remain special but I'm not going to blindly assume that automating the story will have a positive ROI.

Re:Questions to ask yourself when writing tests by andrewflnr on Monday January 01, 2018 @11:56PM (#16049966)
Think about something that necessarily involves hardware interaction, or a GUI, or where all the interesting error cases are non-deterministic (concurrency, network error handling). Ok, on the last one you're pretty much hosed anyway. But we're not all writing nice data-in/data-out apps.
Re:Questions to ask yourself when writing tests by crdoconnor on Tuesday January 02, 2018 @09:10AM (#16051887)
Actually, I probably wouldn't count network error handling, because you can probably use something like vaurien - https://github.com/community-libs/vaurien - to deterministically mimic bad network conditions in an integration test in a fairly reasonable amount of time.

Also, integrating that tool would probably have applications beyond a single story so even if the change takes 5 minutes and making the test work with vaurien takes half a day, it's probably still worthwhile.

Assuming that no tool like vaurien exists, though (and there are plenty of scenarios out there where you'd have to build it from scratch), building the test scenario could become prohibitively expensive.

Re:Questions to ask yourself when writing tests by falsedan on Tuesday January 02, 2018 @05:46AM (#16051137)
Let me help expand your imagination: what if the code is for sshing to prod and needs a ssh-agent or a password typed in? Would you leave those cress on your CI system? Would Security be happy with that?
Re:Questions to ask yourself when writing tests by albertgoeswoof on Tuesday January 02, 2018 @07:26AM (#16051472)
Why would a test build ever ssh to production?

If you absolutely need to test connecting to an external system you should mock it or create an environment for testing against.

Your credentials should also be stored as env variables so the test ones are used only for the test environment.

Re:Questions to ask yourself when writing tests by falsedan on Tuesday January 02, 2018 @01:20PM (#16053923)
Because you're developing automated tools to deploy to prod & you're verifying the correctness of the system end-to-end with a no-op command. The connecting to an external system is not what's under test, and assume I've written unit tests that mock out the sshing, and that even with a staging environment I have to smoke test the new code before switching everyone over to the new version.

Where are those env variables stored on the CI system? In the job config? I would use Vault for this.

Re:Questions to ask yourself when writing tests by KirinDave on Monday January 01, 2018 @10:07PM (#16049557)
That's a decision you can make, but all you're doing is costing yourself more time later for the benefit of less time now.
Re:Questions to ask yourself when writing tests by crdoconnor on Tuesday January 02, 2018 @03:40AM (#16050704)
5x 5 minute manual test < 1 day

Automatd test fundamentalists are subject to the same kinds of folly that other fundamentalists are.

Re:Questions to ask yourself when writing tests by KirinDave on Tuesday January 02, 2018 @12:13PM (#16053293)
And when someone else modifies the code and introduces a subtle bug, will you attribute the costs handling that to this same timebucket?

How many customer minutes lost = 1 developer minute?

That said, I think tests are the most expensive and most brittle way to address this problem. They're necessary, but should be deployed sparingly.

Re:Questions to ask yourself when writing tests by crdoconnor on Tuesday January 02, 2018 @01:19PM (#16053907)
Yes, I would. However:

A) In such cases I think testing against the real thing often has a greater chance of catching subtle bugs rather than the automated test scenario against the elaborate mock which is highly likely to share many of the same assumptions that caused the bug.

B) Code reviews ought to flag that a piece of code that is not under automated test is being modified and appropriate care should be taken (ideally this alert should be automated but I haven't yet reached that level).

I think it pays to approach these things on a case by case basis, and if a pattern of subtle bugs does appear that's a strong indication that you should change your behavior (I'm a stronger believer in bug retros than I am in any kind of testing dogma).

>How many customer minutes lost = 1 developer minute

Is that a relevant question to ask? If you introduce the presumption that an automated scenario test is more likely to catch a bug than a manual test then I guess it's relevant, but honestly for these types of scenario I think the opposite is true.

I didn't mention it before also but if you have manual testers on hand that changes the calculus too. I'd say it's normal for 3-4 manual tester minutes to be equivalent to about 1 developer minute.

As I mentioned above, I really don't think it pays to be a test fundamentalist.

>That said, I think tests are the most expensive and most brittle way to address this problem.

Or a type system fundamentalist.

Re:Questions to ask yourself when writing tests by KirinDave on Tuesday January 02, 2018 @05:04PM (#16056208)
Actually, my personal belief is that people, humans like you and I, are all awful at writing software. We're even worse at enumerating and writing tests.

Miserably bad. Unforgivably bad.

Slowly refining and testing systematizations of correct software building processes is perhaps the most important thing we can do in the first 2 centuries of "software" as a thing.

Because otherwise, all we'll do is continue to wallow in pride and failure, claiming it can't be helped. All the while using language like "case by case," that I have taken to mean: "I will never do that unless you force me to."

Fortunately, I think the scope of failure and fraud in the software industry has grown so late that folks are starting to take correctness as a requirement and not a nice to have. Another Equifax or two and maybe a nice DAO hack or something and folks are going to start saying, "Maybe it's just too bad we all learn to make bad software," turning to new techniques and practices.

Far-fetched? Maybe. But it is happening with AI...

Re:Questions to ask yourself when writing tests by crdoconnor on Tuesday January 02, 2018 @05:58PM (#16056677)
>that I have taken to mean: "I will never do that unless you force me to."

I actually created my own open source BDD framework and a ton of tooling to help automate stories.

19 out of 20 was a pretty conservative estimate of how much I automate - it's probably more like 39 out of 40. I'm a little obsessive about it because I want to dogfood my work properly so I automate quite a few things where the cost/benefit for a normal programmer would seem a bit low.

I'm very cognizant that the industry as a whole is terrible at testing and I'm hoping my software can one day do a small part to help with that.

Re:Questions to ask yourself when writing tests by KirinDave on Monday January 01, 2018 @10:06PM (#16049551)
I wish people would extend this same logic to type systems and formal methods.
Re:Questions to ask yourself when writing tests by mooreds on Tuesday January 02, 2018 @08:15AM (#16051648)
Isn't the wave of FP causing some folks to do this? It takes time for people to change their attitudes. Consider that we're still having this conversation about automated testing, which was promoted by the XP book in the early 2000s.
Re:Questions to ask yourself when writing tests by qaq on Monday January 01, 2018 @09:15PM (#16049362)
I like having tests but I would bet if you audit you production stack it's very likely that a very large portion of it does not really have tests yet is prob most stable part of your stack like Linux kernel etc.
Re:Questions to ask yourself when writing tests by simonhamp on Monday January 01, 2018 @10:20PM (#16049614)
Of course, just because you haven’t personally written tests for parts of a stack doesn’t mean that you shouldn’t write tests for other parts, especially the parts you’ve created.

Also don’t forget that certain things like Linux are extremely battle-tested. It’s generally unlikely that any piece of software you write will receive that much real-world testing, so a growing set of automated tests will help you cover your own arse.

Re:Questions to ask yourself when writing tests by muzani on Monday January 01, 2018 @11:55PM (#16049958)
Perhaps this always applies for manual testing. When it comes to automated testing and something like Android, it can potentially take much longer, especially with the architecture that allows easy testing.