It's more than that. I'm on a mailing list for a social organization that I've belonged to for 20 years, and has been hosted at the same place for all of that time. I have marked hundreds of the messages not spam, and still Google sends about 80% of them to spam.

Google does not care even if you have regular correspondence with an address -- if the server isn't big enough, it's going to spam. The user's wishes or ideas about what is or isn't spam are irrelevant.