First, nobody has an incentive to provide negative feedback except for serious cases (i.e. you cannot work with this person) or for political reasons. It's a prisoners dilemma with indefinite iterations, so each player's incentive is to provide positive feedback.
Second, in my experience, it's only reviewed by one person rather than by a management team, which opens it up to an array of biases on the manager's part. If a manager thinks that someone is a great team-player, and a reviewer says that that person is a petty tyrant, they're likely to doubt the reviewer's motives rather than updating their beliefs.
Peers supply the feedback, but one manager has the time/resources to review it... so basically they can interpret it however they decide.
Your manager can say "based on feedback everyone says you suck" and you can't really do a whole lot to look into it. You don't know where the feedback is coming from, or if it's even valid.
At a smaller company a manager seems more responsible to look at performance democratically. So if they have a personal vendetta and they fired someone for it, it's much easier for other employees and other managers to see what's going on and call it out (so they might not act on it to begin with).
Of course that's not universal, sometimes at a small company you might have one person with consolidated power so it doesn't really matter (and can be much worse). And of course, at a smaller company peer pressure can keep issues under wraps because you're worried about everyone knowing that you're the one bringing it up.
So yeah, a performance review process needs to exist at larger companies because it makes sense logistically... but criticism of that process doesn't mean it should be thrown out entirely, it just welcomes the discussion about how it can be improved. There are middle grounds between strict performance reviews and small-company democracy.