Singer's is a utilitarian framework for arguments about vegetarianism and other animal-rights issues; not an argument for vegetarianism, per se. Animal Liberation is a great, mind-opening read.
> Most people agree that killing primates it morally clearly wrong.
Roughly, you're probably right: killing primates is generally wrong. But it's not always, clearly wrong: killing a primate that's about to kill you, killing primates that are consistently killing other primates, killing one primate to save a hundred other primates, etc., etc. Singer's framework gives us a way to reason-through cases where there isn't consensus.
There is even vast common ground between meat eaters and vegetarians. Most people agree that killing primates it morally clearly wrong.