There was so much polish in other areas that it was odd that the rough edges were never sanded down. The bad AI & especially diplomacy were one of the most immersion-breaking experiences and unlike some of the other tweaks they haven’t been well solved by community modders.
It's funny seeing Civ V so praised, because it was a decisive game when it was released. The AI bugs you mentioned were occasionally annoying, but overall I loved V - the combat changes alone made the game better then IV, but I always felt like a bit of an outcast when people would rag on the game.
I feel similarly about VI now. I really enjoy the depth added to the city changes and how they cleaned up the UI. I hope it gets a similar hindsight appreciation as V.
Let's just pretend BE and 6 never happened.
I'm not sure how else you explain screwups like "will trade 9 gold now for 1 gold-per-turn over 10 turns". It's like they understood the concept of a loan but got the system backwards.
Or military strength judgements, which are blatantly just running on "number of currently active units". You can be feared for having lots of spearmen in the Industrial Era, but the ability to crank out 10 high-end infantry per turn isn't even present in the assessment.
It's a bizarre and embarrassing weakness for a series most people play in single player. These aren't grand strategic errors, they're "make the number bigger/smaller" errors that random modders handle vastly better than 2K.
By the time Civ 5 came out in 2010, most series fans would have been playing fully-expanded Civ 4, with all the rebalancing and new content that entails. Anyone who liked vassal states, Great Persons, corporations, or the complex space and culture victories was going off expansion content.
First-release Civ 5 made some interesting alterations to the series, but had a lot of implementation issues. Unbalanced wonders, AI bugginess, and a late-game slog often made it feel bad regardless of the design choices. BNW-era Civ 5 brought a lot more logic to the tech tree, smoothed out broken features like "one free technology", and made the late game vastly more interesting.
Honestly, I suspect we've only recently gotten to a 'fair' comparison of 4 and 5, with both of them feeling like classics in light of BE and 6.