Having been in construction I call tell you a contractor can build two identical houses in the same neighborhood, with the only difference being the contractor is planning on selling one after it is built, while the other a buyer is exists for the house. The second house will costs at least $20,000 more than the first, the only difference being the buyer will upgrade lights, doors, flooring, countertops and so on. All those little $300 upgrades add up.
This results in fewer people "upgrading" into new construction, which further constricts the "starter home" pool (nobody builds starter homes).
Your comment does not negate the effect that interest rates have had on affordability. The last time interest rates were this high, home prices were significantly lower than what they are now. Wages have not kept up with inflation.