IPv6 was a clean break from IPv4 with a lot of changes because the authors knew that there was a very high 'minimum cost' to any IPv4 replacement. Even if they only added more octets to the addresses, there were still billions of networking devices which would need replacing, millions of people to retrain to some degree, mountains of software to modify or rewrite, and a similar conscious decision to switch. If we had to go through this pain anyway, then the obvious choice was to go all-out designing a greenfield new protocol which solved as many problems as possible and which would last for as long as possible. That's IPv6.
IPv6 is here to stay. It's already 30% of the internet, and has industry-wide buy-in and backing. All OS' support it, Apple even requires apps to work in IPv6 only environments to get on the app store. New IPv6 deployments go online every day.
IPv6 will win. One day. The costs of keeping IPv4 around are high and increasing. ISPs are spending millions on specialized CG-NAT infra to keep their dwindling pool of IPs usable, in return for tremendous capital and maintenance costs, reduced performance, increased support burden from angry customers who had their app broken by CG-NAT. And anyone who wants to buy an IP address has to deal with the price - which just keeps going up.