I say that as a Software Engineer who is 39 and has never been remotely close to affording even a first home.
You may say that it could have increased someone else's stock of housing. And that's true. But if I built it as a vacation home, then if it wasn't going to be used as my vacation home, I wouldn't have built it, so it wouldn't have increased anyone else's supply of housing anyway.
The same is happening even in small town now - a friend bought a place just before Covid for $480k, now it's worth $740k. This is a town of 10k people, in the middle of nowhere. Tiny lot, house is almost 100 years old.
Of course they pocketed a seven figure profit, because of the lakefront property, and the 100 year old wood cabin was bulldozed.
And then the transient occupants party their asses off. And then the Karens pass local ordinances. And then nobody can have any fun.
This is a collective action problem. If one person decides to not move to Undiscovered Gem, Wild West then there is a large number of others that will.
Housing can not be both affordable and a good investment.
Given second homes are an investment, by definition having one means you're helping make homes unaffordable.