In my opinion you're overlooking that a speculator could "engineer" a situation faster than say a local council is able to act / realize what is happening.

For example, a speculator moves into a nice small village, sees potential and brings in a lot of foreign capital in a short amount of time. Buying up houses and outbidding potential new citizens. The price of land is also driven up from this, even if it's only a short term thing.

The damage has been done there and it had little to do with the number of houses. It was just a hostile takeover of a community.