Not quite. In the absence of hard evidence either way (yes, I know, A/B testing -- but there isn't enough traffic on the tarsnap website right now to get any statistical significance for minor changes) website design is purely a matter of taste... and if it's going to be a matter of taste, why not go with mine?
And what you said isn't true, although most engineers I know believe it. It's not A/B testing or bust, and design isn't series of arbitrary aesthetic decisions.
You can also gather (non-numerical) data by going out and talking to potential customers, figuring out exactly what they need, and then making design decisions with that as an input.
That's not entirely true. Design and typography pros have over the centuries developed a series of rules and guidelines for what constitutes good and usable design. These have generally proven to be fairly universal and are more than simply a matter of taste.
1em versus 1.5 vs 2em? That's premature optimization.
And statements like "[l]owered contrast helps readability, but I hate the look of grey text" tell me you're designing for yourself, not for your customer.
That's fine, if that's what you're doing, but I've been assuming the opposite, viz., that your goal is to increase the number of people who pay for tarsnap.