Continuous Deployment at IMVU: Doing the impossible fifty times a day.

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by patio11 on Tuesday February 10, 2009 @06:39AM (#475039)

One of the greatest lines I have ever read on a blog:

It may be hard to imagine writing rock solid one-in-a-million-or-better tests that drive Internet Explorer to click ajax frontend buttons executing backend apache, php, memcache, mysql, java and solr. I am writing this blog post to tell you that not only is it possible, it’s just one part of my day job.

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by plinkplonk on Tuesday February 10, 2009 @10:08AM (#475220)

"writing rock solid one-in-a-million-or-better tests that drive Internet Explorer"

I find this unparseable. (English is not my native language). As far as I know "one in a million" means something like "very rare". Help?

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by briansmith on Tuesday February 10, 2009 @10:50AM (#475293)

"rock solid" is "very reliable" "one-in-a-million-or-better tests" is "tests which fail less than one in a million times".

Our Internet-Explorer-based tests are very reliable; they fail less than once per million executions.

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by sbraford on Tuesday February 10, 2009 @10:13PM (#476440)

While the "one in a million" better is a cool blurb, what does it really mean?

Let's say your team makes 25 commits per day.

25 * ~300 working days = 7,500 commits per year

That would take 133+ years to reach 1 in a million.

The more interesting metric to me is how often the build gets broken.

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by nuclear_eclipse on Wednesday February 11, 2009 @12:19AM (#476577)

The part you're missing is the 15000 tests, multiplied by a new commit every 9 minutes, which in 8 working hours, is roughly 50 commit-test cycles, so 750,000 tests run in a day's timespan...

Edit: of course that assumes a peak commit rate matching or exceeding the commit-test cycle period. The point being that even a considerably low rate of failure in the testing mechanism could manifest itself as a blocked commit-test-deploy cycle at least once a day, hence the importance placed on rock-solid testing systems that should only ever fail when the tested code itself fails.

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by TimothyFitz on Wednesday February 11, 2009 @03:56AM (#476771)

We empirically have on average 70 builds a day. The number is higher than your calculation because we don't all work 9-5, we're commiting frequently from around 8am to 9pm. We also run builds repeatedly overnight to flush out any intermittently failing tests we may have recently introduced. We'll run the builds as fast as they can go from 2am-4am.

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by sbraford on Thursday February 12, 2009 @08:00AM (#478372)

So how often does a commit get checked in that causes a test (or tests) to fail?

It just seemed to me like you were bragging that tests get run over and over again. They only need to get run if any new code is committed, of course.

And what kind of commit is being checked in every 9 minutes? How big is the dev team? Seems like an awful lot of commits. Is each one a full-fledged feature / bug fix for the site, or are many 1-line changes to the code?

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by kragen on Wednesday February 11, 2009 @11:15AM (#477162)

As I read it, the issue he's talking about is that it's easy to accidentally write a test that passes the first 100 000 times you write it, but then fails the next time because of a timeout that was set too low or something like that. A test like that can waste a lot of your time tracking down a nonexistent bug.

It's true that any particular test that spuriously fails one in a million times may never fail. But if you have tens of thousands of tests, and you do tens of test runs per day, you'll have a test spuriously fail once a day or so.

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by ivank on Tuesday February 10, 2009 @10:11AM (#475232)

He means the test is broken if it fails more than once in one million test runs.

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by ivey on Tuesday February 10, 2009 @10:44AM (#475282)

When I say reliable, I don’t mean “they can fail once in a thousand test runs.” I mean “they must not fail more often than once in a million test runs.” We have around 15k test cases, and they’re run around 70 times a day. That’s a million test cases a day. Even with a literally one in a million chance of an intermittent failure per test case we would still expect to see an intermittent test failure every day.

He means that test failures are not acceptable in their culture, and they have a LOT of tests.

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by timf on Tuesday February 10, 2009 @10:16AM (#475239)

In the article it talks about tests that fail only one in a million times (referring to the quality of the test, not the test case being exercised), I think that is what is meant.

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by smhinsey on Tuesday February 10, 2009 @10:10AM (#475231)

i am a native speaker and it's not entirely clear to me either but i would feel safe in venturing to guess that it's a reference to the false positive rate

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by ntoshev on Tuesday February 10, 2009 @09:39AM (#475182)

These are automated functional tests. It would be interesting to know if the author uses unit test too and how he writes tests in general. Also some code:test ratio metrics would be indicative of how much weight does all this testing add.

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by jmathes on Tuesday February 10, 2009 @03:24PM (#475863)

I work at IMVU as well, know the author personally, and can tell you that yes, he unit tests too. In fact, like all of us, writes all code test-first.

For test strategy in general, you need a whole seminar. There's a good blog on testing over at http://googletesting.blogspot.com/ <- go there!

As for code:test ratio, I'm not sure what metric you want, but using just php files, there are ~1k active test files and ~7k php files total. So if you take into account that 1k of that 7k is the test, and some is test-setup and obsolete files we no longer use, it's maybe about 5:1 code:test. However, a lot of the code is third party OS software packages. In-house code is about 1.5:1 test:code, in my experience. These, by the way, are very rough estimates.

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by ntoshev on Tuesday February 10, 2009 @04:30PM (#475994)

Thanks a lot, this is pretty much what I wanted to know. Looks like there is not too much overhead and the results are amazing.

[ Parent | on HN ]

Re:Continuous Deployment at IMVU: Doing the impossible fifty... by edw519 on Tuesday February 10, 2009 @12:04PM (#475460)

not only is it possible, it’s just one part of my day job

A necessary, but not sufficient, requirement for a nimble start-up.

[ Parent | on HN ]