Polyfills are available for most of the usual things and they're absolutely trivial to implement. No one is asking you for IE6 support (if they are, ignore them), but IE11 can do most things decently with minimal effort. As long as a browser is receiving mainstream support and the effort isn't too much, you should be at least minimally supporting it.
Polyfills are available for most things but they’re not trivial, and they have a meaningful cost in two axes: developer effort, and performance. Things like CSS Flex, CSS Grid, SVG, and ECMAScript 6+ functionality can all be made to work in IE (with a few exceptions, like grid auto placement), but only by making things heavier, typically for everyone. Other things that are really nice to be able to depend on like CSS Custom Properties and more recent ECMAScript syntax (e.g. modules) either can’t be used outright, or require much more extensive working around which still restricts what you can do in all the other browsers.
Speaking as someone who has had to maintain IE11 support for a couple of things within the last few years, it seriously is a real drain that contributes more code, bugs (not just for the offending browser) and simple complexity than many realise.
IE is not receiving mainstream support. The effort is too much. You should either ignore it, or, if people are trying to get support on specific issues in it, strongly consider actively scuppering it.
IE is dead. Let it stay dead.
If you don’t have any evidence that you need to support it, I strongly recommend deliberately removing support for it, whether that means just not caring if it works or not or actively breaking it (though please implement that as an IE blacklist, not a certain-browsers whitelist). If you do have evidence that you need to support it, push to remove it, even if you can’t remove it just yet. If you can stop supporting IE, it will make your life easier and it will make my life easier.