The question is whether more progress occurs under the current patent system than would without it (or under an altered system). Are inventors more likely to be properly rewarded for their inventions, thereby motivating them to invent? And does the value of this motivated invention outweigh the value of allowing everyone the ability to freely build on others' inventions?
It is certainly a tricky question to answer, but you can look at the problems faced by countries that don't have strong patent enforcement to get an idea of what would happen if we scrapped the system entirely.
It does seem to me that the situation has gotten out of hand, with completely obvious inventions (especially in the software field) being granted patents, but that doesn't mean that doing away with the entire system is the answer - which your argument here seems to imply.