Re:What you need may be “pipeline +Unix commands” only
by srean on Sunday March 03, 2019 @08:38AM
(#19294370)
> But to deal with Terabytes of data quickly and efficiently, these tools totally break down
Scaling up to 400~500GB of logs with awk and parallel has not been a problem for me. I dont think TB will be particularly hard, especially if one is reasonably proficient with the tools. Of course, if one has the mental bias that one has to throw hadoop or spark at it, thats a significant obstacle right there. Upton Sinclair effect also plays a role -- It is difficult to get a man to understand something, when his salary depends upon his not understanding it.
Of course at some scale simple unix tools become impractical, but usually people reach for flashier tools even at scales where unix tools will suffice.
Scaling up to 400~500GB of logs with awk and parallel has not been a problem for me. I dont think TB will be particularly hard, especially if one is reasonably proficient with the tools. Of course, if one has the mental bias that one has to throw hadoop or spark at it, thats a significant obstacle right there. Upton Sinclair effect also plays a role -- It is difficult to get a man to understand something, when his salary depends upon his not understanding it.
Of course at some scale simple unix tools become impractical, but usually people reach for flashier tools even at scales where unix tools will suffice.