For complex desktop apps I don't really think Java was a viable option until the 1.3 release in 2000 (from memory). You also always had that problem that you couldn't guarantee what version of Java somebody would have installed so you'd have to bundle whatever JRE version you needed into the installer. Deployment was always a challenge.
In the browser Java never really got past 1.1 because Sun and Microsoft fell out so you had to install a separate plugin, which almost nobody did, or just live within the restrictions of AWT.
Couple this with the fact that, as you've pointed out, Java Web Start was so late to the party and it's easy to see why it didn't do better, either on the desktop or in the browser.
Until Android came along I really thought Java was a dying technology platform. Obviously since then its fortunes have significantly improved.
Although what is interesting is that with the Java solutions, it was the user who felt the pain where developers had a decent experience. As the web platform has evolved, users have avoided most of the issues and the pain has been experienced by developers. This would go a long way to explaining why users have never had it so good as they have right now and every developer I know is trying to figure out how to get out of the industry, or at least get out of web dev.
Ever tried to use HP ILO's Remote Console via Web Start? (For additional nerd points, try doing that with Apple M1 machine). Compared to that, the HTML5 version just works.
I recall back in the 2000s there being a few Java apps for macOS that used Cocoa UIs via the Cocoa-Java bridge that used to exist. These tended to be embraced almost as readily by users as full-native Mac apps, and I think that was largely because these apps tended to start up more quickly and fit into the user's desktop better, removing those two barriers to adoption.
Sadly here we are now with bigger computers, better performing JVMs, great Java dev tools, and still losing to the browser due to massive inertia.