News for hackers. Stuff that matters.
Re:A Faster Alternative to Jq by rennokki on Friday March 27, 2026 @06:29AM (#47540939)
> Uses jq for TB json files

> Hadoop: bro

> Spark: bro

> hive: bro

> data team: bro

Re:A Faster Alternative to Jq by anonymoushn on Friday March 27, 2026 @08:44AM (#47542039)
are those tools known for their fast json parsers?
Re:A Faster Alternative to Jq by rennokki on Wednesday April 01, 2026 @05:07AM (#47598546)
If we talk about TB or PB+ scales, then yes.
Re:A Faster Alternative to Jq by anonymoushn on Wednesday April 01, 2026 @11:18AM (#47602072)
Oh, can you post some benchmarks? I didn't know that parser throughput per core would change with the amount of data like that.
Re:A Faster Alternative to Jq by f311a on Friday March 27, 2026 @08:48AM (#47542074)
JQ is very convenient, even if your files are more than 100GB. I often need to extract one field from huge JSON line files, I just pipe jq to it to get results. It's slower, but implementing proper data processing will take more time.
Re:A Faster Alternative to Jq by rennokki on Wednesday April 01, 2026 @05:06AM (#47598544)
More than 100GB can be 101GB, 500GB or 1TB+. I was speaking about 1TB+ files. I'm not sure you can get it faster unless you have a parallel processor.
Re:A Faster Alternative to Jq by eevmanu on Friday March 27, 2026 @11:33AM (#47543996)
made me remember this article

<https://adamdrake.com/command-line-tools-can-be-235x-faster-...>

  Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

  Conclusion: Hopefully this has illustrated some points about using and abusing tools like Hadoop for data processing tasks that can better be accomplished on a single machine with simple shell commands and tools.
Re:A Faster Alternative to Jq by rennokki on Wednesday April 01, 2026 @05:05AM (#47598535)
This article is good for new programmers to understand why certain solutions are better at scale, there is no silver bullet. And also, this is from 2014, and the dataset is < 4GB. No reason to use hadoop.

The discussion we had here was involving TB of data, so I'm curious how this is faster with CLIs rather than parallel processing...