Patents are of limited duration because the tradeoffs of unlimited patents are so horrific. If we accept a billion dollar drug must enter the public domain, clearly copyright should also be limited just as it was proposed in the US constitution. However, because a tiny minority has a huge benefit and society does not really notice the difference you get the modern mess of unending copyright.
You haven't diminished the ability for 350 million people to do things, practically, by shaving 1 cent off of them. But adding the ability a million dollars provides one individual to do something cool with 1 million they couldn't do before has increased the overall capabilities of everyone.
In essence, you've just described Kickstarter's business model.
Basically, even if individual bacteria are unnoticed enough of them can kill you.
If tiny minority + 'society' is the entirety of the system, then that's true, but there are also plenty of players who lose out due to restrictive IP regimes—and it's hard to quantify the extent of those losses. (Whether or not the benefits they would reap from looser IP are appropriate or fair is beside the question for utilitarian computations.)
What would be the point of a patent system with unlimited duration? If we wanted that, we could just have companies not reveal their inventions in the first place
But if laws are proliferating and regularizing instead of standing still or being abolished, and one assumes that elected representatives are acting on the will of the people, it probably does.
There's lots of controversy over how to improve copyright / patent law, but not very many people in governments in the EU, US, China, Japan, Australia, &c are talking seriously about just burning the whole copyright / patent system to the ground. At least a subset of the countries in the groups listed are generally understood to have representative governments.