Your users are going to primarily be using a single browser (per device) and expect each device to be, at least, consistent with itself. This is an area I feel devs make the largest mistake: they want browsers to conform to one another. That means they'll break default functionality to get there! I'm of the opinion the browser should change. That allows users of the browser to adapt to the new default (because every site will then be consistent for the user!)
Functionality can differ between browsers [0] and your IE users will expect the numeric input field to function like it does in IE. Not like it does in Chrome/FF/Safari. Meanwhile, your Chrome/FF/Safari people will expect it to function how it does by default in their browsers. You shouldn't override or change this functionality because it breaks user expectations. Overriding IE because it is the "odd one out" will break expectations for users using IE!
It doesn't matter so much if the experience is different for each user so much as the experience is consistent for each user within their environment.
[0] https://css-tricks.com/numeric-inputs-a-comparison-of-browse...
If all these fields had nice clean ways to style them so they looked the same cross browser (even with different defaults and behaviour), then there would be less people using these giant scripts and frameworks.