What you describe is exactly what's done by most software. In fact, there are often several versions of each function, for each variety of CPU instructions supported. Take a look at x264 or libvpx for an example.