Re:Ask HN: Why is A/B testing unacceptable for novels but en...
by webmaven on Tuesday February 22, 2022 @09:56PM
(#30437044)
A/B testing just isn't the right tool for the job. The combinatorial explosion of things that can be tweaked with small variants, not to mention major changes, is for most practical purposes infinite. Readers also self select to a large degree, some changes to a novel can result in a work that appeals to completely different groups of readers!
What many authors do is have a group of 'early readers' that they know and trust to give them actionable qualitative feedback ("I didn't really feel that decision by the character was consistent with what we know about them", or "I get you're trying to establish the setting, but the action and conflict doesn't start until chapter 3, you need to bring it forward, or at least do some foreshadowing"). This is much more like agile software development that revolves around close collaboration with a customer, and sometimes an author's agent is also an early reader.
That said, publishers also send out hundreds or even thousands of early (usually edited but not always copy-edited) copies of a novel to a much wider group to solicit feedback (and generate buzz). Such Advance Review Copies, or ARCs (and their electronic equivalents eARCs) are often sent out looking for bug reports or pain points ("Hey this unnecessary rape scene isn't cool and doesn't even advance the plot") and some indication as to what sort of readers will most like it, which is often really helpful when a work doesn't fall neatly into a particular subgenre or subverts a subgenres tropes and cliches (you can think of this as the publisher getting data on "people who like X also like Y" or "This is Space Opera for people who don't like Space Opera!"). This stage is more about priming (ARCs go to reviewers as well) and optimizing the top of the funnel than doing major iteration on the book itself, and collecting endorsements from other authors or celebrities to feature on the cover, but detailed feedback if provided won't be discarded, though it will be filtered by the editor.
So, you see, within the constraints of novel publishing and the long lead times to get books printed (12 months from when the editor accepts the manuscript!) Novels actually do incorporate a lot of feedback and iteration, just not the sort of micro optimization that is the hallmark of A/B tests.
What many authors do is have a group of 'early readers' that they know and trust to give them actionable qualitative feedback ("I didn't really feel that decision by the character was consistent with what we know about them", or "I get you're trying to establish the setting, but the action and conflict doesn't start until chapter 3, you need to bring it forward, or at least do some foreshadowing"). This is much more like agile software development that revolves around close collaboration with a customer, and sometimes an author's agent is also an early reader.
That said, publishers also send out hundreds or even thousands of early (usually edited but not always copy-edited) copies of a novel to a much wider group to solicit feedback (and generate buzz). Such Advance Review Copies, or ARCs (and their electronic equivalents eARCs) are often sent out looking for bug reports or pain points ("Hey this unnecessary rape scene isn't cool and doesn't even advance the plot") and some indication as to what sort of readers will most like it, which is often really helpful when a work doesn't fall neatly into a particular subgenre or subverts a subgenres tropes and cliches (you can think of this as the publisher getting data on "people who like X also like Y" or "This is Space Opera for people who don't like Space Opera!"). This stage is more about priming (ARCs go to reviewers as well) and optimizing the top of the funnel than doing major iteration on the book itself, and collecting endorsements from other authors or celebrities to feature on the cover, but detailed feedback if provided won't be discarded, though it will be filtered by the editor.
So, you see, within the constraints of novel publishing and the long lead times to get books printed (12 months from when the editor accepts the manuscript!) Novels actually do incorporate a lot of feedback and iteration, just not the sort of micro optimization that is the hallmark of A/B tests.