The first re-flight of a booster was in 2017. The practical science of rocket reuse is only four years old. During that time they've flown ~40 missions with reused cores with zero failures. One Falcon 9 has been reused 10 times. My point is that there is no reason to assume that we've reached the limits of reuse in such a short period of time, so it doesn't seem that aggressive to assume that there is room to improve by ~10x
There's an enormous gulf between asserting this and the engineering to realize it. Maybe Raptor isn't as reliable as they hope. Maybe 3mm steel is too delicate so they have to stay with 4mm which makes them overweight. Maybe the thermal protection system is too fragile or doesn't provide enough protection. Maybe their novel alloy can't meet its targets. Maybe the tower catch won't work so they need legs which will add too much weight. Etc, etc.
All of those are extremely nontrivial issues, there's a lot of risk because of how ambitious what they are attempting is. I hope they succeed.
This is the biggest risk in the program, along with how many times they can reuse each piece, which is related. Last I looked Musk was projecting using the boosters hundreds of times and the 2nd stage dozens, both obviously very aggressive projections. If the real numbers are lower then the price per flight rises.
As far as I know we don't have any independent way of knowing whether these projections make sense or not. I hope he's right. Even if he's a little wrong and the flight cost was $20M instead of $2M that would be a great thing. But he could also be very wrong and it might not work at all.