News for hackers. Stuff that matters.
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by johndough on Monday January 19, 2026 @08:14AM (#46678625)

    (>= 6 months of 1 gig of data per day)
You can parse JSON at several GB/s: https://github.com/simdjson/simdjson And you could scale that by one or two orders of magnitude with thread-based parallelism on recent AMD Epyc or Intel Xeon CPUs. So parsing alone should not pose a problem (maybe even sub-second for 6 months of data). We would need a more precise problem statement to judge whether horizontal scaling is needed.
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by anshumankmr on Tuesday January 20, 2026 @12:17AM (#46688178)
> https://github.com/simdjson/simdjson

Was not aware of this but seems it is not there natively in Python,but seems cool. Will try out in future.