The worst-case performance of validating a block full of SegWit transactions, which is the only way you'll hit 4MB with SegWit, is actually a lot better than the worst-case performance when validating a block full of non-SegWit transactions. The original design of Bitcoin had an unfortunate flaw where the cost of transaction validation could increase quadratically with transaction size. SigWit fixes this to be linear. (As it happens, Bitcoin Cash copied this fix, which makes it just as incompatible with existing wallets as SegWit transactions; arguably more so in fact, since by the time that deployed a lot of wallet software had implementations already.)