Forget about the philosophical "because you didn't buy that" or the various decent counter arguments. What it comes down to is that publishers purchase very narrow licenses of an author's output. One may specifically buy print rights, and only print rights for a specific geographical region like English USA. Or just the audio rights for a region. The same publisher often does not own both print and audio. So the big publishers that deal mostly in print get very defensive when sales of a copy they had no part in might eat some of the sales they might otherwise get. That's what you see in this case.