The main thing that can be annoying is figuring out module exports from a js library that you want to use (which is probably easy enough if you have a modern javascript background). I haven't found a library that's stumped me yet, but some are more annoying than others. Jetbrains have a tool called Dukat which will supposedly one day automate this, but for now it's not there.
Distribution size can be an issue, but it's not terrible. My current (moderately complex) app is floating at around 2MB. I haven't tried splitting it up, though that is possible.
I don't buy into sharing source files instead of clean separation via Web APIs, the additional tooling complexity isn't worth the pain.
I did vote against stuff like GWT already several times, and every time it has been proven to have been the right decision, when things went hot.
Fair enough, that's a different argument from the one you posed upthread though. The ecosystem and tooling are very much there.
Happy to disagree on architectural merit of the approach. Personally I'm done with javascript/typescript for any future products/teams I build, and am glad to have a viable solution.