"A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system."
> The first rule of distributed systems is don't distribute your system until you have an observable reason to.
“You can have a second computer once you’ve shown you know how to use the first one.”
–Paul Barham
If 1 machine postgres suffices, stick with it. If it does not, would buying more processors/memory/disk fix that? If so, stick with it.
The big data stuff is when you really have too much data to deal with in one box and literally cannot get round that.
The tradeoffs are (my limited experience): cost in terms of buying a stack of boxes. Cost in terms of learning a massive new software stack. Cost in terms of "it scales almost infinitely but each node processes so much less".
I gave a link before, I hope that showed something useful. Here's another, see article and paper https://html.duckduckgo.com/html?q=scalability%20%22at%20wha...
Does that provide (at least the beginnings of) an answer?
a) most companies deal with small amounts of data. Small can mean dozens of megabytes to dozens or hundreds of gigabytes. A single well provisioned server will typically be able to handle that very well. Also an SQL database can do a great deal if you know what you're doing.
b) inappropriately used big data frameworks are expensive performance killers. https://adamdrake.com/command-line-tools-can-be-235x-faster-... for example.
c) Good quality programming, as in understanding the machine, memory layout and why it matters, and a good understanding of algorithms (and a hefty dose of common sense), will often yield you more speedup than buying almost any number of new machines.
c) Hiring is often driven by fads and companies often don't like being told 'you don't need this roomful of servers', they like to waste money, so maybe do learn them (the profligacy with money is likely to be coming to an end with the economic damage of covid).
Takeaway: brainpower will get you much further than horsepower