It seems a CI system should be able to work the same way and cache the dependencies just like a local build.

If it does, then the long compile times are almost never encountered for neither developers nor CI. So are they really problematic?