It's not that every individual has to agree not to kill you. It's that we have to agree that punishment is appropriate for killers. Getting specific - it's incoherent to say it's immoral for humans to kill rabbits but moral for foxes to kill rabbits. To have a consistent moral philosophy, you have to either punish the foxes and try to prevent them from killing the rabbits, or you have to admit that killing rabbits isn't _really_ a problem.
Consider an infant - an infant is basically an amoral agent - it doesn't understand killing. If a infant kills somebody with a gun, we don't say "oh well, he doesn't understand killing" - we say "why the heck did that baby have a gun? Where are his parents?" We expect moral agents to prevent amoral agents from causing harm. So why aren't you trying to prevent the foxes from killing the rabbits?
It's been many years since my collegiate philosophy course that focused on ethics in society - largely using Peter Singer's works as focal points of discussion - but could you try and reframe your idea? It doesn't seem like reciprocity has much bearing on this topic.
It fails Kant's categorical imperative. Animals aren't going to believe that eating meat is wrong - and if we tried to prevent animals from eating meat, we'd kill a lot of them - not to mention it'd just be ridiculous. There's simply no way to include animals in universal law and maintain reciprocity. It's an internally inconsistent ethical philosophy, and obviously so.
That's to say - people make fun of ethical vegans for the same reason they make fun of libertarians - clear gaps in the philosophy they refuse to acknowledge.