Vanity publishers would tell every author their work had great sales potential, then charge them $2000 for $500 worth of editing, printing and marketing - so authors would not only fail to make money, they would actually make a large loss.
As to whether either form of self-publishing 'scam' or not, that depends on the expectations of the author. I think it's always been common to publish just to be published with no expectation of making money, hence the 'vanity'. But I have no data to back that up.
Today you can decide how much your editor is worth. If your grammar and spelling is good you can pay less, or if you know it is bad (like me) you can pay extra until the quality is where you want it. You might even have a friend who will do the early editing for free (a trained editor shouldn't be wasted on spell check duties, but they are probably worth it once you think the book is done for final tweaks) Whatever this investment is, you can limit the costs.
Academic presses are also mostly very low volume. But academics don't expect to make a living from selling books (at least not directly - books help to establish reputations which can help you in the academic job market).