Obviously, driving requires attention and is always one's main focus. Having driven various sizes of vehicles, it's quite clear that this amount of attention varies. For example, the sheer amount of work required to drive a 26 foot box truck gave me a healthy respect for truck drivers. On the original topic, I'd imagine a lot of RV drivers don't pull over or are actively hostile to being passed because they're so overwhelmed they don't want to pile on any more requirements. And while this means that they should be taking the opportunity to stop and recover, try telling someone who is drowning to stop thrashing.
Piloting a 2500 lb manual coupe barefoot in a non-wooded area on an empty road requires the lowest amount of work I've experienced. Driving that same car while having to continually gauge how hard the person in front of you is applying the brakes takes much more. Braking extra and creating more buffer room gives a slight reprieve, but at the ailing speeds ruins views tend to go on non-straight roads you'll inevitably catch right back up.
Just keeping a car on a road with no traffic takes a fair amount of focus: for evidence, just see how many people don’t do it and plow into objects that are alongside the road.