CGNAT specifically means you can't have even temporary peer-to-peer connections, e.g. non-server multiplayer games generally won't work. And forget about trying to host anything, dynamic DNS services can't help you here. That to me is a much bigger problem than IPv4 in general being a bit slower.
v6 also has better measured performance on webpage load times. Perhaps "pages load slightly slower than they could do" isn't a show-stopping problem, but faster would still be better, right?