Before I started writing Flutter I had the same feeling. My first reaction to the Flutter design was "haha, it's not gonna work". But when I tried to make my first app I was shocked how good it was.

Since then I learned that most users do not give a s...t if it's pixel in pixel perfect copy of native components. Especially when Flutter team makes it literally pixel in pixel perfect to the native UI. I mean, it really doesn't matter in 99.99% cases in my experience (I have around 12 apps in Flutter atm).

So I see this Flutter design feature as huge upside, not a downside.