If anything inference will probably get worse. Typically in ML constraining your architecture (which is what you're doing when you make a deep narrow network out of a shallow wide network, or rnns or grus/lstms, or transformers out of fcnns) yields inference and training performance benefits.