Re:The shittiest project I ever worked on (2013)
by teddyuk on Friday May 24, 2019 @07:29AM
(#20000391)
This really is true, I worked on a project where we were meant to be getting hourly files into a data lake, the files so small we couldn't reach the recommended size of 256mb per file (compressed parquet in azure adls) - the files were like 1 mb each - a years worth of data was tiny and the processing overhead ridiculous
[ Parent
| on HN ]
Re:The shittiest project I ever worked on (2013)
by jjoonathan on Friday May 24, 2019 @07:49AM
(#20000535)
If your workflows don't bog down your servers, add big data technologies until they do!