News for hackers. Stuff that matters.
Re:FizzleFade by afpx on Tuesday August 29, 2017 @05:46AM (#15123171)
Recently, I started helping out data engineering teams improve performance of their big data processing pipelines.

Man, was I shocked. Very smart, highly educated, mid-level and even senior software engineers seem to know very little about bits these days. When they'd run into a memory issue, their natural response was to just spin up a few more servers and throw another terabyte of memory at the problem. Makes sense, I guess - until their CFO saw their pretty exponential curve in infrastructure costs.

Re:FizzleFade by hasenj on Tuesday August 29, 2017 @06:07AM (#15123241)
It's lucrative for SaaS providers to convince developers that they don't need to care about performance. Just throw more memory at your problem. We'll be there to provide you all the hardware you need. wink wink.
Re:FizzleFade by Illniyar on Tuesday August 29, 2017 @06:12AM (#15123266)
And did knowledge of bits help solve the memory issue?
Re:FizzleFade by MichaelGG on Tuesday August 29, 2017 @08:07AM (#15123697)
I've done stuff where memory usage shrank from 30-60GB to under 2GB, and load times went from many minutes to essentially instant (mmap on the bits Vs needing to build objects).
Re:FizzleFade by afpx on Tuesday August 29, 2017 @06:28AM (#15123325)
That was implicit. Of course it did! :-) And, cut server costs in half, too.
Re:FizzleFade by auggierose on Tuesday August 29, 2017 @09:42AM (#15124177)
I like bits, too, but cutting it in half means it is still exponential ;-)
Re:FizzleFade by afpx on Tuesday August 29, 2017 @09:53AM (#15124254)
To be fair, the savings are way above 50%, honestly. But, I didn't want to claim a bigger number without digging in more, and 50% was a very conservative upper bound. But, yeah, you're right.