Google knows, for example, that I get the Vagrant-DigitalOcean mailing list mail, and open it maybe a handful of times a year. If they are downgrading that mailing list's reputation based on that they're 100% wrongly reading my intentions into their data. That's Google's fault (even if they're externalising the costs ont ot he mailing list).
It becomes a problem when you pump lots of email into a good sized provider, and your open rates are poor. Google knows more about what mail you read than you do.