> How about not showing it at all until the person has seen a dozen or so pages?
We actually tested that as well (though I think we did it after 6 pages). The conversion rate was of course higher than showing it immediately, but the users didn't open or click any more frequently. (Hence, showing "immediately" on first page load is better business-wise.)
> Does your A/B test detect how annoyed the readers are?
Haha, that is indeed why I pushed against this in the first place (if I hadn't, we wouldn't even have tested it so thoroughly). But the fact that such "annoyance" didn't show up in any metrics ultimately won me over.
One thing to keep in mind is that I've only tested with non-technical users, who I think are in general much less annoyed by this sort of thing than the typical HN person.
As many have pointed out, most are unlikely to subscribe to the newsletter/install the app/whatever before they've even seen the site.
It seems like the least annoying way to do this would be to detect multiple visits (preferably over multiple days) and then ask for the signup. If the person says no, don't show it to them again for a very long time, if ever.
"We also ran this A/B test over multiple weeks to see if it had an effect on return frequency. It didn't.)"
Does your A/B test detect how annoyed the readers are?