I think the argument is more along the lines of "if you look after the pennies, the pounds look after themselves," not the slight straw man of "every single little accessibility improvement benefits every single user." If you have accessibilty at the heart of what you're doing then it will almost inevitably improve the product for everyone to some extent. My argument would be stronger, in that I think there's a very high probability that it will improve the product significantly for a large proportion of your audience.