Mesh vs mesh collisions will never be faster than primitive collisions. Also, simplified collision geometries do not produce glitches in the same way mesh vs mesh collisions do. At most you may have a collision between two primitives where the models don't actually seem to be touching. But this can easily be circumvented. In fact, I would argue that a lot of the physics glitches we've seen are probably the result of using mesh colliders when a simplified geometry would have sufficed.