For complex desktop apps I don't really think Java was a viable option until the 1.3 release in 2000 (from memory). You also always had that problem that you couldn't guarantee what version of Java somebody would have installed so you'd have to bundle whatever JRE version you needed into the installer. Deployment was always a challenge.
In the browser Java never really got past 1.1 because Sun and Microsoft fell out so you had to install a separate plugin, which almost nobody did, or just live within the restrictions of AWT.
Couple this with the fact that, as you've pointed out, Java Web Start was so late to the party and it's easy to see why it didn't do better, either on the desktop or in the browser.
Until Android came along I really thought Java was a dying technology platform. Obviously since then its fortunes have significantly improved.
Although what is interesting is that with the Java solutions, it was the user who felt the pain where developers had a decent experience. As the web platform has evolved, users have avoided most of the issues and the pain has been experienced by developers. This would go a long way to explaining why users have never had it so good as they have right now and every developer I know is trying to figure out how to get out of the industry, or at least get out of web dev.
Ever tried to use HP ILO's Remote Console via Web Start? (For additional nerd points, try doing that with Apple M1 machine). Compared to that, the HTML5 version just works.