At the same time, giving up a market will give another competitor a golden oppurtunity which could come to bite them. And countries could probably somehow force Google to continue operating Youtube there by threatning other Google products.

I kinda doubt it. Youtube's "moat" against competition is the fact that storing, processing, and serving video is insanely expensive. Google can only afford to do it to begin with because they are the world's largest digital advertising company.

It would have to be another player willing to invest a ton upfront, and continue burning ludicrous amounts of capital over many years before they could actually compete

I mean the competitor could also be large company, for example, dailymotion.com could take over from Youtube without any major hassle. And EU would probably love it since dailymotion is local company.