I recently asked about the "unlimited vacation" policy and received a vague answer about "establishing expectations with your manager" and then was told they went with someone else.

I thought about if I should ask or not but I honestly don't understand how such a policy could be good for employees so I wanted an idea of the culture that allows it. If that question contributed to them not giving me an offer then good riddance so I don't regret asking.

No need to keep asking. If your get is telling you that something's not quite with this "unlimited vacation" policy -- it's because it's a sham, basically.

And everyone at the company knows it -- including and most especially, upper management. Because that's after all where it originated.

EDIT: 'get' should have been 'gut'.

Completely agreed. An interaction like that during or after an interview would be a big red flag that it's not a place I want to work.