The point is not for students to learn how to follow an algorithm (or master doing it quickly), but that they learn why the algorithm works and how to interpret the result.
The algorithm matters and I argue that once students can naturally derive the algorithm on their own, there's no place for human calculations. Computers and calculators should be used. We're no longer living in a society where students will have to worry about "what if you don't have a calculator on you". It seems as weird as memorizing unit conversions to me.
What's interesting is that the purpose of implementing the common core standard was exactly what I am arguing for. A shift away from algorithmic memorization to a number sense way of doing math. The failure lies with the teachers not being able to teach well much more than students not learning.
I agree with your general point that math curriculum builds on itself and if students fall behind it can make things worse and worse for future classes. But in general I think, at least when I was in school, there was probably too much focus on algorithms and not enough on conceptual understanding. I think part of the reason is that algorithms historically were necessary to know before calculators were ubiquitous, and also I think algorithms are much easier to teach and evaluate so there's probably a tendency to focus on them for that reason.