And even then you'll often end up significantly worse off than if you wrote the assembly by hand.
A run of Intel's compiler on the C versions of our DSP functions resulted in a grand total of one vectorization, which was done terribly, too.
I am told that Fortran does better than C here; there is a reason it is still widely used in the scientific computing community, after all.
This is also part of the reasoning behind C99's new restrict keyword.
If you don't write the compiler, you have to make assumptions about when and how it can/will use those instructions, and often you assume wrong, particularly across compiler upgrades.
The first reason is the whole category of optimizations that the compiler is worse than a human at (like register allocation) or cannot do effectively at all (messing with calling conventions, computed jumps).
The second reason is more subtle: in any case that you abstract yourself from some part of a problem, you inherently create a less efficient solution.
For example, intrinsics mean that you don't have to manually allocate registers. But this also means that if your algorithm uses too many registers and it would be more efficient to modify it to require fewer (and thus not need spills), you will have no way of knowing such a thing. By insulating yourself from that layer of complexity, you've also limited your ability to make higher-level optimizations that improve lower-level performance.
This applies on practically every level possible: any method of abstraction, no matter how well designed, will always in some fashion reduce the maximum performance you can achieve. Of course, this doesn't mean abstraction is bad--it provides an often-useful tradeoff between developer time and performance.
You can get 5x, 10x, 20x, or more performance increases just by using the vector instructions given to you by the CPU. Until a magic compiler appears that can make proper use of them (read: never), hand-coded assembly will be critical for almost any application for which performance is critical, especially multimedia processing.