There were literally per-seat licenses at $5000 and $10000 at one point. There were free implementations around the time when Java was new, like GNU Smalltalk, but it would have been better if those hadn't existed, because they didn't convey the benefits at all.

LightTable had YouTube to enable people seeing the benefit of the environment. Smalltalk could have really used that at the right time. That still wouldn't have helped, because the Smalltalk community didn't understand the network-effect benefits to the same degree almost any programmer understands them today.

Smalltalk could have been Java. Sun actually approached the owners of the VM to ask if they could use it for their early set top box project. They were rebuffed and went on to write what was to become the JVM. (Smalltalk becoming Java might well have resulted in huge early Internet security disasters blamed on it, because of the difficulty in securing the system, and the preference developers had for code signing as the primary security mechanism.)

Meanwhile the part of the Self group still at Sun created Pep, which was a Java on top of Self that was way faster than any other Java. But it was a quick hack with a few serious limitations.

Animorphic, the StrongTalk company, did a more polished implementation of Java on top of their VM technology and got bought by Sun to turn that into the HotSpot VM working together with the part of the Self group that had stayed at Sun all along.

Around 2000 Sun allowed the Self project to be restarted until they killed it again in 2006, at which time they open sourced StrongTalk (which had been hidden since Animorphic had been bought).

It did. Much later than the introduction of the JVM, however.