Rasterization has some niceties for things like cache coherency. Since you're rasterizing a triangle at a time, all with the same shaders, textures, and buffers, SIMD techniques used in GPUs is very effective. But when you have ray-tracing, any ray can hit any triangle. If you cast 64 rays and each one hits a different triangle, you lost your entire parallelism. You possibly have to give up on SIMD if the material models between the triangles are different.

For this reason, and for the reason of real-time ray budgets barely approaching 1 ray per pixel at 1080p on the highest cards, ray-tracing tends to mostly be used for specular effects at this time.

Even in non-real-time workloads, this was a massive time sink until ray sorting and batching entered the common practice, collecting rays that hit a single coherent area to be processed at once. And the current RTX model has no strong support for ray batching.