If you look at discussions, you'll see that the actual decline started earlier, by some opinions as far as the end of season 4. The problem is that back then, it was not as clear that the series was declining - it could've been a super interesting new story thread or a new intrigue starting out. But now, with the knowledge that it leads nowhere, you can clearly see the problems starting out.

Also, of course, it's less fun watching the stuff now, knowing that it will slowly fade out into a bad ending.