Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by bell-cot on Tuesday March 08, 2022 @10:18AM (#30600794)

> I have used those before, and yet I still had those characters appear in data...

Oh, yes - which is why I emphasized "is". But ASCII text is easy to test for, which lets you fast-track into exception handling - "Tell Sales that Customer data is not as represented", "Trouble-shoot internal data source", etc.

(My experience is that substantial Customer data is never, ever as initially represented. Nor as represented after you point out the first set of issues with it. Nor as represented after you point out the second set of issues. Nor as...)

[ Parent | on HN ]

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by IanCal on Wednesday March 09, 2022 @07:34AM (#30613752)

> Oh, yes - which is why I emphasized "is". But ASCII text is easy to test for, which lets you fast-track into exception handling - "Tell Sales that Customer data is not as represented", "Trouble-shoot internal data source", etc.

Oh with that I mean the ASCII control characters appearing in inputs. So some columns would have record end markers in for example.

If I'm able to make everyone dealing with the reading and writing add specific characters to be used for start/end/etc I'd rather just tell them to swap to a parquet reader unless they've got a really good reason.

[ Parent | on HN ]