The same argument against increasing the block size but even more so. It takes time for blocks to propagate across the P2P network, which puts smaller miners at a disadvantage by increasing their orphan rate, which leads to miner centralization.

I think that argument is even stronger against lower block times because latency (network, validation, etc) rather than bandwidth contributes the most to block propagation delays.