It's not that every individual has to agree not to kill you. It's that we have to agree that punishment is appropriate for killers. Getting specific - it's incoherent to say it's immoral for humans to kill rabbits but moral for foxes to kill rabbits. To have a consistent moral philosophy, you have to either punish the foxes and try to prevent them from killing the rabbits, or you have to admit that killing rabbits isn't _really_ a problem.
Consider an infant - an infant is basically an amoral agent - it doesn't understand killing. If a infant kills somebody with a gun, we don't say "oh well, he doesn't understand killing" - we say "why the heck did that baby have a gun? Where are his parents?" We expect moral agents to prevent amoral agents from causing harm. So why aren't you trying to prevent the foxes from killing the rabbits?