Negative rights also “infringe” on others freedom in the same way you describe: they limit what others can do. The “oblige others to inaction”. The trade off is they (usually) get the same protections.