Because other miners are required to use a higher timestamp for their block to be valid, they have no ability to correct my outlier time. You have essentially allowed me to put the blocktime permanently forward by the allotted wiggle room, and also you have allowed me to consume all that space you allocated for miners who have clock skew.
It's overall a small attack, but highlights that even tiny decisions can have consequences that impact security. Bitcoin accounts for this by requiring that timestamps be greater than the median of the past 11 blocks, and that's enough to prevent one evil miner from forcing the block times forward for all blocks.
But even if all miners decide to do as you say and always pick a date slightly in the future but inside the wiggle room, all that they achieve is a one-time very slight difficulty decrease, but by the next difficulty adjustment that's already lost because by then both the start date and end date of the period are offset by the same amount into the future.
There are consequences to having blocks that are permanently set forwards beyond a one-time difficulty decrease of 0.6%.
My reasoning was roughly as follows:
Miners are incentivized to pick very large timestamps, because the longer it takes to mine X blocks, the easier becomes the proof of work they need to solve, giving them more block rewards and transaction fees in the long run. But if they want the network to accept their blocks, they can't pick timestamps in the future, so the best they can do is pick the current time as their timestamp.