It depends on the hardware. There may be offloading for h.264, VP8, for different profiles and for encoding or decoding. But not all combinations are available on all machines. So software decoding may still be needed.

Now software decoders are often written with some hand-optimized assembly loops, but those optimized variants don't cover all possible target platforms and not every single loop is hand-optimized. So everything that's either a plain-C fallback or just hasn't been optimized can benefit from compiler improvements.

Nope. They could be GPGPU accelerated, but if you aren't rendering to screen then it's not typically graphics processor territory.

Plus modern codecs are quite linearly data dependent which is a bad for for a GPU.

But aren't movies rendered to the screen?

My understanding is most smartphones at least have hardware decoding for H264.

Exactly, a GPU needs to have specific hardware (a dedicated ASIC) for H264 acceleration. It's not accelerated by the usual GPU hardware, which is sufficiently general-purpose that an accelerator running there would not be codec specific.

Arm even sells such a solution, there is a line of Mali video processing units.