News for hackers. Stuff that matters.
Re:Monitoring Sucks. But Monitoring as Testing Sucks A Lot More by btilly on Thursday July 19, 2012 @11:00AM (#4266272)
At Devopsdays I listened to a lot of smart people saying smart things. And to some people saying things that sounded smart, but really weren’t. It was especially confusing when you heard both of these kinds of things from the same person.

Experience has taught me that if someone who is clearly smart, who clearly says smart things, says something that sounds dumb to me, it is worthwhile for me to not just dismiss that. Instead I need to examine my preconceptions for why I disagree, and why they came to the conclusion that they did. Until I have satisfied myself that I understand both why they thought as they did, and why I disagree, there is a good chance that I'm missing a valuable lesson.

This example is a case in point. Clearly it is from the point of view of a web company. The advice offered is not for all environments - there is a world of difference between a case where downtime means someone doesn't see a picture for 15 minutes and one where someone dies.

Now about unit testing and monitoring, let me give an example. I know a company (which I can't name) that releases multiple times per day, and releases every change as an A/B test. This is important. If they release a change, that works exactly as designed, that hurts conversion by 10%, THEY WILL KNOW. (You need significant traffic to follow this strategy, they have that.) This is important. There are a lot of trivial changes that could move the needle 10% without your realizing, and you don't want to move it in the wrong way.

In fact, if you look at the dollar values, a bug that causes 1% of pages to crash which unit testing could catch is simply not as important as a bug that hurts revenue that the A/B test could catch.

But it gets better. If you have 0 tolerance for web pages crashing and have monitoring in place to catch it (I know people who have all crashes email key developers), then you'll catch a lot of bugs that you would catch with unit tests, AND you'll catch bugs that you SHOULD have caught with unit tests but messed up on the test. Which, then, provided more value, the unit test or monitoring?

You want the unit test. You don't want to be catching stuff after you roll out. And one of the automatic questions when you do catch it should be, "How could we have caught that with an automatic test?" But having smart monitoring is more valuable to you than the unit test.

Re:Monitoring Sucks. But Monitoring as Testing Sucks A Lot More by Terretta on Thursday July 19, 2012 @11:42AM (#4266513)
The A/B test or phased rollout test works if you have great telemetry (NewRelic, etc) and sufficient traffic that results are statistically sound.

A low traffic site working on MVP releases can synthesize both the OP article and the comment above by "unit testing" the essential functions of the web site from users' browsers point of view. Think Selenium, Watir, PhantomJS, and the like.

http://jquery.bassistance.de/webtesting/presentation.html