Regarding your points:
1. Amazon takes action when a brand complains, even with only a single complaint. Sellers have to prove the complaint is wrong to be allowed to continue to sell the product, or in some cases to be reinstated. Usually this is done by providing an invoice which Amazon verifies.
2. The enforcement is usually at the seller level, not the listing level.
The owner of No Starch Press shows up at a lot of CS shows and other shows. Next time you are at one, seek him out for a chat. Tell him Anazon takes action after a single complaint. Be prepared for him to disabuse you of your misconceptions.
Source: I heard his story first-hand at PyCon.
I very much doubt Amazon is rejecting DMCA notices, and suspect the story is more complicated than that. I've heard from many sellers that it's easy to file complaints, and I know many sellers who have gotten false complaints, which is also what you'd expect if it's super easy to file complaints.
Knowing how Seller Performance works, they aren't about to close off a listing that is doing $nn-K revenue per month because of a couple of copyright reports. The economics simply don't make sense for Amazon to poison the inventory of a seller because of a couple of reports, to fix this problem Amazon truly needs to figure out a way to verify the validity of inventory sent in to their warehouses.
That's a question that I don't have an answer to, but I do hope that someone from Amazon is really working on it. Inventory commingling and a myriad of other issues present in the processes of selling on Amazon are the reason that I decided to not pursue FBA further.