DMCA requests usually don't result in a website shutting down unless the webmaster is non-responsive and the claimant makes further moves beyond the first request.
The easiest solution would be to create penalties for false takedowns of legal content, but this would likely just result in Google being sued for trillions of dollars by the media cartels to remove such restrictions under the argument that Google is aiding in piracy.
That's when ContentID and the non-DMCA takedowns started happening.
You hit on this in your first sentence. And this whole arrangement with Google is specifically to side-step those penalties. Your 'easiest solution' is for Google to return to DMCA takedown requests. I agree. I want that. I think most consumers and content creators would.
But Big Media doesn't.
I mean, how fast would Google move if a non-responsive take down resulted in the youtube.com domain being seized?
Instead, Google would rather have made their own job easier by standardizing 'requests.' There's nothing in the DMCA specifying that some standard, machine-readable format be used. To get the media companies on board with keeping the takedown request process as automated as possible, they compromised.
Even if Youtube was in the right, it would take years and multiple appeals to get to that conclusion and then Viacom still appealed so at some point Youtube just decided to settle, it was cheaper.
I believe that means that in the current legal environment it is not cost efficient to provide free hosting of third party videos, ad supported, without draconian/significant number of false positives content identification system that is made available to the biggest potential "suers". Thus, even if a competitive service would appear, if it needs to be profitable using the same format it would likely end up implementing a similar (or worse) system. But I'd be happy to be proved wrong.
Whoever is in charge of this situation dropped the ball out of fear. It cost Google plenty because all videos should have ads unless you are on a paid account.
The rights holders may end up playing the two services against eachother, and use the more draconian policy of one as arguments for making the other more friendly to their interests.
They have a lot more skin in the game than users do.