Patents have now ended up enforcing exactly that which they were supposed to oppose - corporate monopolies on thought.
Just to name a few examples... The often quoted example of steam engines and the patents of James Watt: the power output of steam engines (an objective, quantitative measurement) stagnated during the period Watt's patents were valid. Only after they expired, steam engines started improving.
Another example: the contributions of the Wright brothers to modern aviation. The Wrights filed for a lot of patents but most of their inventions fell into obscurity as aircraft manufacturers opted to use patent-free inventions from the Wrights' competitors, such as De Havilland's rear stabilizer design which is still common in modern aircraft.
I'm not convinced that the patent system has ever worked to genuinely promote innovation and invention. There are numerous examples of how it has done the exact opposite by stagnating a field of science and technology (e.g. video codecs today).
Several high profile inventors such as Thomas Edison have spent the latter days of their lives as businessmen and lawyers protecting their patents and financial interests, not inventing and innovating.
The Wright brothers tried to hinder development of airplanes, but De Havilland managed to circumvent their patent. Creating entirely new technology in the process.
Watt on the other hand managed to protect his steam engine patent and went on to invent a copying machine with the money.
And Edison, mediocre inventor who luckily made some patent money early with the quadruplex telegraph. Then he could concentrate to his true calling of enslaving scientists.
I'm half joking here. But we really can't know how history would have turned out if patents would not exist. So far I'm not convinced with any of the alternatives. Even with the glaring flaws of the current patent system.
Furthermore, a lot of the accusations levied against him are just outright false. For instance, the "electrocution of an elephant":
https://en.m.wikipedia.org/wiki/Topsy_%28elephant%29
Edit - more about Edison: http://www.nj.com/middlesex/index.ssf/2013/02/oatmeal_comic_...
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1589712
The Wright patent certainly was influential and used all over the place. Their litigation did force the invention of alternatives. Innovation through forced workarounds had long been one of the rationalizations of patent systems. However, very few of them were non-infringing. Indeed a criticism of their patents was that they were way too broad and covered everything, but courts time and again gave them broad scope because they were deemed that inventive.
As an aside, before anyone brings it up, the related story that the Wright patent held back the US airplane industry has also been shown to be a myth, as demonstrated by industrial data fun the period:
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2355673
If you think work is stagnating in video codecs, it's mostly because the low hanging fruit had been picked.
Examples of the patent system creating benefits (as well as costs) can be readily found by searching ssrn for keywords around patents. Focus on studies that use empirical data. Also if one study looks too convincing, look for others that cite it, there may be one that busts it :-)
A lot of the conceptions people have about patents are based on falsehoods and uncritical acceptance of what somebody on the internet said. This need not be the case when we have a number of relevant studies to look at.
Video codecs certainly have had the low hanging fruit picked, but if you look at the situation with video on the web today (ie. consumer level tech, not state of the art), the situation is not much better than in 1998 (to exaggerate a little): we're still using badly implemented browser plugins. We use Flash video because Adobe et al have the patent issue covered. Yes, HTML5 video exists but the actual codecs are such an IP and patent minefield that out-of-the-box support for video (esp. in open source browsers) isn't really ready for prime time.
Google has tons of patents but their secret sauce is private. Intel has patents on their devices, but their semiconductor processes are closely held secrets.
Companies have their cake and eat it too.
The more persuasive argument is that patents encourage invention by rewarding it. If it costs 1 billion to develop a product that create 10 billion worth of value but the inventor won't be able to capture more than 500 million of the value, the inventor won't spend the money.
The other good argument that nobody seems to make is just a fundamental property right. You create the invention, you own it. Real estate property was first doled out like that. First to claim it, keeps it.
Only certain ideas (production methods implemented in shops or factories, for example) could be hoarded. Other ideas had to be disseminated in order to create economic value from them, like the classic example of a "better mousetrap." There's no way to sell a better mousetrap without revealing how it traps mice. Anyone who buys one can disassemble it and then build their own.
Without patents, small inventors are defenseless against bigger organizations, which can replicate the invention, manufacture it faster, and put more marketing dollars behind it.
Patent protection, like many federal laws, is a balance. On one side, it does not allow the replication of inventions without permission of the inventor, to protect small inventors. On the other side, as you mention, it requires public disclosure of the invention, so that method inventions that would normally be secret forever can eventually be used by the public.
The reverse is also true.
With patents, small inventors are defenseless against aggressive litigation from deep-pocketed bigger organizations. If an inventor is small in a strict sense, he certainly doesn't have a few million dollars to spare to defend himself in court.
It's not about patents, it's about the terminally broken system.
We have little guys (patent trolls) suing the largest most profitable companies on earth.