You don't need to scale. You're not Google, Amazon or FB to have a billion daily active users.

And if you manage to make it to that scale, you can certainly hire engineers to refactor those systems. Fix the problem only when you encounter it.

Parsing 1Tb of logs daily with a 14 day retention? A desktop with a NAS attached can do that.

The corps that developed these fancy tools and then open sourced them and labeled them hipster ish names? Of course they want engineers out there to use them, it's called vendor lockin.

Yes, you are right. But there are a lot more usecases, e.g. imagine you run a website for a (small or large) government, or a website for a popular television show. Or perhaps you are a developer in a telecom company in some country. Scale is everywhere.

> you can certainly hire engineers to refactor those systems.

Wouldn't it be better if those engineers used standard tools like Hadoop?

I worked for the largest online news site of Switzerland. We were nowhere near the scale where such tools would provide significant benefit over the "boring", "outdated" technology we we using. We also had a fraction of the operational cost than our much much smaller competitor within the same publisher that went full hipster on their tech.

Thing is, it for a lot of use cases it is a whole lot harder to get to large scales from a business and product perspective than it is for good engineers to adapt to it.