Academic presses are also mostly very low volume. But academics don't expect to make a living from selling books (at least not directly - books help to establish reputations which can help you in the academic job market).
Vanity publishers would tell every author their work had great sales potential, then charge them $2000 for $500 worth of editing, printing and marketing - so authors would not only fail to make money, they would actually make a large loss.
As to whether either form of self-publishing 'scam' or not, that depends on the expectations of the author. I think it's always been common to publish just to be published with no expectation of making money, hence the 'vanity'. But I have no data to back that up.
Today you can decide how much your editor is worth. If your grammar and spelling is good you can pay less, or if you know it is bad (like me) you can pay extra until the quality is where you want it. You might even have a friend who will do the early editing for free (a trained editor shouldn't be wasted on spell check duties, but they are probably worth it once you think the book is done for final tweaks) Whatever this investment is, you can limit the costs.
Low selling authors have been a big thing since always, because there are a huge number of markets that are small enough that it was (and is) not unusual for publishers in smaller markets to print on the order of a few hundred books per run for unknown authors.
E.g. in Norway (where Kjell Hallbing/Louis Masterson is from), 10k sold used to mean you were a big deal, and high up on the bestseller lists.
I suspect there is now a middle ground that didn't exist before: In the 1970s you were either selling >10,000 copies, or you weren't a published author. 'Self-publishing' had a reputation as a scam to extract money from naive would-be authors. (I'm not sure what the academic book market was like at the time)
It's only with the rise of ebooks and print-on-demand that niche, low-selling authors have become a thing.