Marathons are harder for people who run them in 4 hours (assuming they tried as hard as they could). They are on their feet longer, and have to coerce their mind to push that much longer. It is just a different race for mid and back of the packers. Further, training is much the same. You might run 50-60 miles in a week because you can go faster. A slower runner might dedicate 30 or 40% more of their time to achieve that many miles. It takes the same mental effort to run at a 4 RPE (RATING PERCEIVED EFFORT on a scale of 1-5) as it does for an elite. The elite just has genetics on their side. I used to feel much the same way you do, but truly those "joggers" slogging out a 4 or 5 hour marathon giving it their all after months of running 9+ hours per week. They are runners too.

Ultras are also far less populated by folks with an elitist running mentality. It is more about being in nature and a personal journey, which is why Ultra appeals to a certain crowd. Olympians and the guy down the road have more in common with their running experience than not.