Not that much harder than writing hand optimized assembly.

There are a few more quirks to keep in mind and you need to have the memory lay-out and access patterns down otherwise it will not give you the boost you expect.

In a way I really love CUDA for this reason, it allows me to re-cycle all my old optimization skills and do something useful with them.