As others have said, the light time is an issue, but it's also not a constant data stream - we uplink data directly to the rover, but can only get data downlinked by relaying it through the MRO and Odyssey orbiters. That happens 2-3 times a day (or sol).

The Mars rover is using radio signals to communicate with Earth. Radio signals are part of the electromagnetic spectrum and thus travel at the speed of light. Even at the speed of light, it's going to take some number of minutes for radio signals sent from Mars to reach Earth. Unless those hackers are inventing the warp drive, you can't get much faster than that!

Isn't is mostly a bandwidth issue right now? I mean, physics prevent us from currently improving the latency, but we could improve bandwidth for sure. Right now it's fairly low due to distances and power for transmissions.

Reading the NASA site just now the MRO transmits from 400Kbps-5Mbps with a max window of about 16 hours a day. The rover has about 8 minutes per sol to transmit data to the MRO at a rate of about 128Kbps-256Kbps. If the rover goes direct to earth then it drops to 300bps-32Kbps.

I seems like at the very least we could improve by increasing the number of satellites to build out full planet coverage and also keep a transmission point in view of earth at all times. Yes, not cheap, I know.

The light time is a fundamental driver here. See more at https://en.m.wikipedia.org/wiki/Interplanetary_Internet

I think most people wouldn't be very upset if the constant HD stream from a Mars rover is 5-15 minutes delayed.

The real issue is the cost, bandwidth and infrastructure in place to make that happen. Scientists would rather want to use bandwidth, power, launch mass and real-estate on the rovers for science instruments - orbiters around Mars which relays data must be upgraded to handle more bandwidth, the DSN on Earth that receives the data can't be dedicated fulltime to one data stream, etc.

(That said, the new laser communication systems NASA is experimenting is pretty awesome, and will greatly improve bandwidth for space missions)

As I understand it the satellites used as relays are in quite low orbits which limit the amount of time available for rovers to upload. A communication satellite in areosynchronous orbit or at a lagrange point could give more upload time. But presumably that would require more capable transmitters on the rover.

Twitch plays Rover? :)

Rescue Rover? :) https://archive.org/details/msdos_Rescue_Rover_1991