Re:21 GB/s CSV Parsing Using SIMD on AMD 9950X
by zeristor on Friday May 09, 2025 @11:45AM
(#43938128)
Why not use Parquet?
[ Parent
| on HN ]
Re:21 GB/s CSV Parsing Using SIMD on AMD 9950X
by mcraiha on Friday May 09, 2025 @11:54AM
(#43938218)
Excel does not output Parquet.
Re:21 GB/s CSV Parsing Using SIMD on AMD 9950X
by speed_spread on Friday May 09, 2025 @01:07PM
(#43939013)
True. But also Excel probably collapses into a black hole going straight to hell trying to handle 21GB of data.
Re:21 GB/s CSV Parsing Using SIMD on AMD 9950X
by hermitcrab on Friday May 09, 2025 @01:53PM
(#43939420)
Excel .xlsx files are limited to 1,048,576 rows and 16,384 columns.
Excel .xls files are limited to 65,536 rows and 256 columns.
Re:21 GB/s CSV Parsing Using SIMD on AMD 9950X
by mihular on Friday May 09, 2025 @02:18PM
(#43939654)
21GB/s, not 21GB ...
mawk would handle a 21 GB csv (or maybe one true awk) fast enough.
Re:21 GB/s CSV Parsing Using SIMD on AMD 9950X
by buyucu on Friday May 09, 2025 @02:13PM
(#43939614)
Excel often outputs broken csv :)
Re:21 GB/s CSV Parsing Using SIMD on AMD 9950X
by hinkley on Friday May 09, 2025 @06:30PM
(#43941535)
I have been privileged in my career to never need to parse Excel output but occasionally feed it input. Especially before Grafana was a household name.
Putting something out so manager stops asking you 20 questions about the data is a double edged sword though. Those people can hallucinate more than a pre-Covid AI engine. Grafana is just weird enough that people would rather consume a chart than try to make one, then you have some control over the acid trip.
Re:21 GB/s CSV Parsing Using SIMD on AMD 9950X
by constantcrying on Friday May 09, 2025 @02:49PM
(#43939884)
Or HDF5 or any other format which is actually meant to store large amounts of floating point data.