That’s basically because ISPs are a semi-natural mononpoly without competition there is really no incentive to do better. Heck Comcast acts like a monopoly in my area and they are competing with AT&T who offers better service - 1Gbps up and down [1] vs 1Gbps/35Mbps Comcast, at a lower price and Google Fiber in some parts of town.

[1] I’ve got to give a shout out to AT&T and BackBlaze, I just changed my Plex Server over to a new computer and my entire 1.7TB collection was uploaded in less than 3 days. I was getting speeds between 500GB per day and 1Tb+ a day.