News for hackers. Stuff that matters.

A comment on Command-line Tools can be 235x Faster than your Hadoop Cluster (2014). Parent comment

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by awild on Tuesday March 08, 2022 @02:32PM (#30605266)
Just a heads up, pandas to Csv is laughably slow. You're literally better off converting the pandas df to arrow and then serialise to Csv with arrow, though the format is quite limited with arrow.
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by NavinF on Wednesday March 09, 2022 @04:48AM (#30612697)
Well at least it’s better than numpy’s csv functions. For my use case (~4 million images, one row per image, 2048 fp32 columns for features extracted with resnet), parsing/encoding csv with pandas was 10x faster than with numpy. Of course serializing the arrays to/from npy was 10x faster than any csv library.
[ Parent | on HN ]