The distinction to me is: Are they building and saving some data structure that they've created based on my emails which is designed to target ads towards me, but could be used by some nefarious actor to learn a lot about my life?

The spam filter (hopefully) probably doesn't store that much about me, but an ad targeting bot could be a lot more problematic.