Loosening IP restrictions would have first-order benefits to a lot of people who could now do things with existing IP that were previously impossible.
The downsides would be delayed and second-order: a significant portion of people, who currently invest in developing IP with the expectation of "owning" the resulting product, would reduce their level of investment. Why invest in R&D if you can just wait for someone else to find a successful idea and then copy them?
Why spend a billion dollars testing a drug if its market price is immediately dropped to generic level by cheap manufacturers? The only option left is secrecy, which is worse for society than patents.
Furthermore the arguments used for it, can also be used in regards to any investment with high costs and ease of replicability. Even something as simple as a rice farm would be able to justify a monopoly over selling rice in a region in that manner, after all, it's a lot cheaper to simply import it from elsewhere once the market demand for it has been established by the farm.
I'm saying, for some investors (I did say a significant portion, perhaps the quantity can't be verified), IP ownership is a factor in their decision making. Are you saying it doesn't for anyone? It's a different thing to argue that the "aggregate" level of investment across the whole economy would be unchanged. That's a different issue, and anyway the nature of any investment would be affected by changes in the law.
On the second point yes, the argument can be extended to absurdity but the scope of the patent system is a judgement made by legislators, which should (absent corruption) be based on a judgement of its relative benefits/disbenefits.
[0] Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial https://www.bmj.com/content/363/bmj.k5094