This is vastly oversimplifying the problem, the difference between IPv4 and IPv6 is not just the format of the address. Different protocols have different features, which is why the sockaddr_in and sockaddr_in6 types don't just differ in the address field. Plus the vast majority of network programs are using higher level abstractions, for example even in C or C++ a lot of people would be using a network library like libevent or asio to handle a lot of these details (especially if you want to write code that easily works with TLS).
Yeah, the big issue is that any code that took addresses from user input had to do validation to make sure addresses were valid, in allowed ranges, etc.
While the sockaddr struct allowed to to abstractly handle v4/v6 socket connections, there wasn’t a clean way to do all of that additional stuff and IP address logic leaked into all kinds of software where you wouldn’t first expect it.
Something as simple as a web app that needs to inspect proxy headers would even have it.
It also didn’t help that it became practice to explicitly not trust the addr resolution offered by the sockets API because it would do unexpected things like resolving something that looked like an integer to a uint32 and then a 4 byte V4 addr.
Of course, code that operates on packets, in the TCP/IP stack of the OS would have still needed to be rewritten. But that is far less code than "every application that opens a socket".
Of course, this only applies to code that uses IPs only to open connections. There's lots of application code that does more things with IPs, such as parsing, displaying, validating etc. All of this code would still need to be rewritten to accept IPv6 addresses (and its much more complex string representations), that part is inevitable.