You could use appropriate tools like valgrind and compile with ubsan to make this a non-issue.

So I'd happily take more performance for code without UB trading off slight inconveniences for debugging UB issues.