Slapping some scipy, sklearn, or pandas stuff into Django was a pretty workable solution even back when. Just want to sing the the praises. I've slayed a lot of normal size (as opposed to big) data problems with Django/Flask. Rub some JS charting on top of it and people thought you were a wizard.

Oh it was workable. Until it wasn't. I specifically remember extricating a high throughput web service from pandas. Having to scale made that stuff harder. If it was some app that a few thousand people were using, no problem.

> If it was some app that a few thousand people were using, no problem.

You’re right, scale is absolutely an issue for companies that sell software or that are very large.

Also important to consider that most companies use software internally instead of selling b2b or b2c. Fewer than 500 businesses in the US employ more than a few thousand employees* (https://fortune.com/fortune500/2020/search/?f500_%20employee...) and there are over 17.5 million businesses (https://www.naics.com/business-lists/counts-by-company-size/).

* I may be wrong in my assumption that the Fortune 500 companies are the largest employers

Even then, scaling is an almost self-solving issue, as it usually comes with money attached, which buys quite a lot of engineering hours spent on scaling things. Overscaling in advance on the other hand with all of its associated complexity and slowdown killed many, many companies.

Inability to scale because the application was fundamentally unscalable has also killed many companies. And for those it didn't kill it significantly hurt their earning potential (and ultimately everyone's payout).

There's a balance and it's more delicate than people give it credit these days. Don't get me wrong though... over-engineering and/or premature optimization is just as bad. But there is an opposite extreme as well.