That's definitely the hard part of switching to Bayesian methods. But the problem doesn't go away if you use p values, it just gets ignored.

Typically people doing research have prior information about what they are researching. E.g. previous studies have found effect sizes to be in some interval.

You can fallback on an uninformative prior or other tricks if you really want to model the idea that you know literally nothing about what you're researching. But that should be very rare.

You just make priors up, just like the 0.05 limit is made up. Making up a prior is basically asking the question: How likely does it seem that X has Y effect? Without having any data specific to that question.

Our 0.05 p-value limit is effectively just answering that question with a fixed 5%, no matter how ridiculous the proposition is.