From a distance, I think highly of Harvey Mudd, but I don't know their grading policies. I doubt they grade harder than Caltech, since HMC's branding is all about "we want to help you succeed" and Caltech is more of "we're going to beat you up [intellectually]."
However the graduation rate tells perhaps a (slightly) more nuanced story about rigor. HMC, despite beimg more expensive to attend than Caltech, had a lower 2023 graduation rate of 91.5% vs Caltech at 93.7%
it's not that different, and probably you can construct error bars that enclose both of these bounds within plausible parameters, but it suggests that there is a somewhat higher premium for failure of those actually accepted at HMC than even the mighty Caltech, and a greater willingness to uphold standards at the expense of angering (relatively) wealthy parents.