I sold one of the 3 houses I've owned & sold to someone using a buyer broker. The broker reached out to negotiate the fee, and we settled on 2.5% (a bit too easily, I suspect I could have pushed it down to 2%). The last house I sold we almost sold via a buyer agent, and they did the same thing.

They are doing work, so the real question is whether you feel they are overcharging or not for whatever it is that they do.

Also, I think you meant to say "you lose practically every buyer WITH a buyer agent". This has not been my experience.