It's even worse! Even after you have completed the transaction, you don't have the counter-factual and never will: Could someone else have done it better/faster/for more?

I would think a referral is only really excellent if from a person who had to do many transactions recently, and sampled at least 2+ realtors. And that's something that's quite hard to come by.

It's just like any hiring process: get referrals from people you trust, look at prior work (previous purchases/sales in this case), learn the right questions to ask when interviewing.

Yes, you need a trusted word of mouth referral, good agents live on them.

See, there's this education and licensing process, with a Trademarked(TM) name and everything: REALTOR(TM).

Unfortunately, this process is what we have now.