News for hackers. Stuff that matters.
Re:The shittiest project I ever worked on (2013) by teddyuk on Friday May 24, 2019 @07:29AM (#20000391)
This really is true, I worked on a project where we were meant to be getting hourly files into a data lake, the files so small we couldn't reach the recommended size of 256mb per file (compressed parquet in azure adls) - the files were like 1 mb each - a years worth of data was tiny and the processing overhead ridiculous
Re:The shittiest project I ever worked on (2013) by jjoonathan on Friday May 24, 2019 @07:49AM (#20000535)
If your workflows don't bog down your servers, add big data technologies until they do!