It's not as contingent as you think. We normally model the supply curve as a line going up and to the right, indicating that price increases stimulate the creation of additional supply. The demand curve has the opposite slope and effect. These curves form an "effective field theory". They explain what's happening at a high level. In reality, supply and demand curves aren't necessarily linear, and at small scales, they can wiggle around a bit. This wiggling is where your "it depends" factors come into play.

It's disingenuous, however, to argue that this wiggling invalidates the crystal clear overall direction of the supply and demand curves or somehow invalidates the idea that the way to increase the amount of affordable housing is to increase the amount of housing.