In general you talk to other known experts in the field and see what they say.

If you don't find an expert in the field and you publish a crappy book you tend to get a number of reviews (hopefully before you send it out to retail) that your book sucks.

When selling books about fake evolution you're not looking for an expert in the field, you're looking to see if you have an audience that would eat it up.

This is what credentials used to signal, but I’m not sure they’re as reliable anymore…

That used to be at least somewhat true about scientific and technical books. Major publishers protected their brands by hiring editors with some subject matter expertise. Of course, we understood that those books could be outdated, incomplete, or biased but for established publishers the quality was generally fairly good.

You as a buyer have publisher's reputation as a filter. There's still chaff but it's a bit easier to avoid.