There were a ton of stories during the season how the ratings were falling off a cliff. Although not bad in the larger picture like you point out, it would seem there is a general downward trend of viewership for NFL games.
Power Rankings: The reasons why NFL ratings are down: http://www.si.com/nfl/photo/2016/10/20/nfl-television-rating...
NFL ratings are still way down and getting worse: http://awfulannouncing.com/2016/nfl-ratings-are-still-way-do...
10 theories why NFL TV ratings are down: https://www.bostonglobe.com/sports/patriots/2016/11/05/theor...
This is exactly what I did - following through halftime, I didn't exactly watch Lady Gaga's halftime show, and just never came back.
Was in shock later when others told me that the Patriots had won...
There's also Game Pass where you pay something like £150 a year to stream all games live (although they blackout the ones shown on Sky) and on demand which also gives condensed and highlights version of the games.
Super Bowl 2016: How Many Countries Will Watch Carolina Panthers vs. Denver Broncos?: http://www.ibtimes.com/super-bowl-2016-how-many-countries-wi...
Though the Super Bowl lacks the global reach of the World Cup, it still has a growing audience outside the United States. More than 170 countries and territories around the world will have the opportunity to view a live broadcast of the NFL's championship game between the Carolina Panthers and Denver Broncos at 6:30 p.m. ET.
So still huge ratings and only down 600k on last year, which isn't bad when there's over 110m viewers.
Secondly although it was a pretty thrilling final quarter and OT at the half it looked like it was all over so I wouldn't be surprised if a lot of people turned off (as it was nearly 2am for me I nearly did).
Finally, these are only ratings for Fox - I'd be interested to know how it did globally.