Re:Command-line Tools can be 235x Faster than your Hadoop Cl...
by johndough on Monday January 19, 2026 @08:14AM
(#46678625)
You can parse JSON at several GB/s: https://github.com/simdjson/simdjson
And you could scale that by one or two orders of magnitude with thread-based parallelism on recent AMD Epyc or Intel Xeon CPUs. So parsing alone should not pose a problem (maybe even sub-second for 6 months of data). We would need a more precise problem statement to judge whether horizontal scaling is needed.
[ Parent
| on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl...
by anshumankmr on Tuesday January 20, 2026 @12:17AM
(#46688178)
> https://github.com/simdjson/simdjson
Was not aware of this but seems it is not there natively in Python,but seems cool. Will try out in future.