You're still not doing apples-to-apples comparisons. The source that you cited says 12th–19th-century life expectancy at birth was 25–40 years when infant/child mortality is included, and 55 years when it is excluded. There has not been any “4x increase” in this time period. (Even if you start in the bronze age, life expectancy at birth has not increased four-fold.)

Since the time period you chose, life expectancy at birth has increased by about 40 to 55 years. Near-elimination of infant mortality accounts for 15–30 years of increase, while increases in adult life expectancy account for an additional 25 years.

North America has a life expectancy of ~80 years. I eyeballed it so closer to 3.2x the 25 year life expectancy in the 12-19th century. What am I missing?

Yes, if you start from a base of 25 years (which is the extreme low end of the range from your source), then we have 30 years of increase from reduced infant/child mortality and 25 years of increase from reduced adult mortality.

Can you really look those numbers and say that one of them is highly significant while the other "has not changed much"?

Translating this into percentages or "fold" increases is misleading because you are using a different denominator for the two increases. E.g. you are claiming that reduced infant mortality is a 2.2-fold increase (55/25) while reduced adult mortality is "only" a 1.45-fold increase (80/55), even though the number of years gained is almost the same.

And if you use the other end of the 25–40 year range, then reduced infant mortality is "only" a 1.38-fold increase (55/40).