You're spot on. One optimization is that once you are above the highest pixel of the map, you can assume you won't hit anything else. Another is calculating if the Earth's curvature will move pixels out of your way before you hit them.