Yup, it's not an exact tradeoff, and near-0 it breaks down entirely. At zero it's worse than that, because of the diminishing block subsidy schedule, having zero fee pressure will reduce the incentive for mining, and therefore security.

However, miners can't simply apply the limit to themselves without a consensus rule. There is code in Bitcoin Core to apply a soft limit, but as long as there is any fee pressure, ignoring transactions leads to a disadvantage compared to other miners, so in practice all miners tend to always use the maximum permitted block size. Most proposed schemes for increased or variable block size still enforce the same size across all miners for this reason.