Re:A lot of complex “scalable” systems can be done with a si...
by roel_v on Monday December 30, 2019 @06:37AM
(#21912646)
If there are floating point numbers in those csvs, I sped up a system like that 10x just by writing a custom (Java equivalent of) atof() that didn't do variable decimal separators and scientific notation. That's not even counting the improvements in I/O speed from the size reduction. Any system that works from CSV's is going to be slow. I don't know what sort of computations you're doing of course, but I did all the work on a laptop, couldn't be bothered to scale it out after the improvements I made in the first pass. How much of your code spends its time in I/O (including conversions) vs actually calculating?
[ Parent
| on HN ]
There are no comments on this comment yet.