This video details why: https://youtu.be/azI3nqrHEXM
The short version is there is an exploit to meet standards by increasing footprint that is easier to implement than to make vehicles more fuel efficient. That exploit needs to be nerfed.
However, the government screwed up the scaling factor and it basically became impossible to build a small truck, while larger trucks had much easier-to-meet requirements. That led to automakers basically abandoning the passenger car market, and upsizing trucks significantly.
Ford left the third generation Ranger to rot for nearly 15 years because people kept choosing the ever-increasing-in-size F-150. Chevy discontined the S10 back in 2004 in favor of the larger Colorado.
Customer preferences were already clear.
https://www.thedrive.com/news/small-cars-are-getting-huge-ar...
Furthermore, vehicles with higher fuel efficiency tend to have smaller engines and thus also have shorter noses and more sloped hoods which make them safer for pedestrians.
The only connection with the CAFE regulations you can make is that they are obviously not strict enough. Especially when it comes to trucks and SUVs. And I fully agree with this. Stricter CAFE regulations would help not only with emissions but with pedestrian safety.
The main reason for the difference between a 1990s f150 and 2020s f150 is that people have been preferring to buy bigger trucks and Ford has catered to that preference by making the trucks bigger CAFE regulations be dammed. To their credit Ford has done some very good r&d work, trying to keep their ever larger trucks relatively fuel efficient, such as by their very well publicized conversion to aluminum. But they could and should be doing more and the government should be doing more to force them to do more.