They were pretty popular actually. The problem was that browser makers started trying to kill applets off very early. Netscape supported them, but Microsoft / Sun had a huge falling out and they ended up pushing ActiveX very hard as a replacement. The dominance of IE assured that ActiveX replaced Java applets for a while, and then MS fell out of love with ActiveX too, so HTML+JS was all that was left.

Applets were very much a victim of various power struggles within the browser industry, combined with Sun's general lack of competence on the desktop - for instance, their online upgrade engines have always sucked. Though in fairness, nobody got that right until Chrome.