Nah, your concerns are 100% reasonable - they just operate on a different context. On Earth, latency is king. On Mars, especially until the Primary Mission is complete, it's all about risk mitigation. Since we're light-minutes away from Earth, a few frames of latency is nothing. At the same time, you want to avoid breaking your $3B machine, which is hard to operate given the time-of-light delay and comms limitations. Just a different set of tradeoffs. IIRC they first tested on-device deep learning for hazard avoidance in Curiosity, but don't quote me on that.

-Worked at JPL for a few years and have dozens of friends, a few in the vision system.