This is absolutely a problem. The buyers agent is a gatekeeper. They will not show your property if the commission is that low.

Source: I used to be a real estate agent. I left the industry because of BS economic incentives like that and others that incentivize you to not put your client's best interests ahead. That and the ripoff racket that is most brokerage licensing requirements and the arrangements enforced by law (that is heavily influenced by the NAR).