There are many failed attempts to draw UI oneself. But ultimately, it is because development cost (you need to implement a lot of widgets, supporting new OS capabilities in on-going basis). If Google can support these development cost, it can be done.
Accessibility issues can be solved for these toolkits. OS often exposed some kind of the API to achieve that. For web, you certainly can have hidden elements to support accessibility. Again, it is just a matter of development cost.
As you said, nobody should seriously mistake Flutter app with semantic web. Most of the SPA online will break if you style CSS differently or disable Javascript, this is just the world we live in. The missing of semantic information for Flutter app should be treated as the generic accessibility problem, not some resistance to the semantic web vision.
But I must say as an app programming platform, flutter is brilliant. The ability to pass around statically typed classes representing everything from widget to text style to animation is so fast and logical, that it actually made me enjoy frontend programming. No need to mess around with html+js+css+some macro language to glue it together.
Did I mention i run the same flutter app on android, ios, web and linux box? I just wish they would use something like kotlin instead of dart, but still, it's well worth it.