It's by definition anti-competitive. Its entire purpose is to reduce competition so that the inventor can profit from it. Even if inventors were required to license their inventions to other people for a truly fair rate it would still be anti-competitive since it would increase the costs (and thus barrier to entry) of competition. The inventor still enjoys a significant advantage in the market (not having to pay the license fees).

It's fair to ask if it's bad for there to be an official form of anti-competition, but call a spade a spade.

One could easily make the argument in this context by this part: "by securing for limited times to authors and inventors the exclusive right to their respective writings and discoveries."

The argument here is that the only ones that should benefit from this are the authors and inventors, and they and only they are granted "exclusive" rights. No one else. In fact, I'd find that fair. The implication being you can't sell or transfer those rights to someone, and those rights are only granted to the authors and inventors.

Which leads us to this question:

"exactly how is it anti-competitive to disallow other companies to use the said patents unless they agree to pay for the rights?"

It's not, as long as the one doing the license is the author or inventor. One should not be able to transfer those rights to a third party, as it's explicitly an exclusive right.