Same way you don't render the whole world each frame: https://en.wikipedia.org/wiki/Potentially_visible_set

Also 3D data structures, which http://realtimecollisiondetection.net/ is still my go-to book.