Re:Data Science at the Command Line, 2nd Edition (2021)
by synergy20 on Sunday May 05, 2024 @08:03PM
(#40269799)
Thanks. I use 'rg' a lot but it finds part of the json text, not sure how 'jq' can be used with that since you're not finding a full json object in general
Re:Data Science at the Command Line, 2nd Edition (2021)
by faizshah on Sunday May 05, 2024 @08:19PM
(#40269889)
Oh I see, so what I do is I use rg -l aka —-files-with-matches. To produce the relevant files and I use the filepath as the key for the next part of the processing pipeline.
So the pattern is to reduce the candidate set of files with rg or find then extract the relevant parts using xargs jq then pipe to jq -s to produce the dataset.
Re:Data Science at the Command Line, 2nd Edition (2021)
by synergy20 on Sunday May 05, 2024 @10:32PM
(#40270613)
That makes sense, thanks!
Re:Data Science at the Command Line, 2nd Edition (2021)
by rahimnathwani on Sunday May 05, 2024 @11:28PM
(#40270879)
If you write a book I will buy it.
Re:Data Science at the Command Line, 2nd Edition (2021)
by wwader on Monday May 06, 2024 @11:23AM
(#40275751)
You might want to look into using jq --stream together with inputs/truncate_stream/fromstream and friends if you want to use jq with large inputs. Not a speed daemon but will probably use a lot less memory.
Re:Data Science at the Command Line, 2nd Edition (2021)
by faizshah on Monday May 06, 2024 @05:25PM
(#40279682)
Thanks, if anyone else is interested there is an explanation of this feature here: https://subtxt.in/library-data/2016/03/28/json_stream_jq And: https://github.com/jqlang/jq/wiki/FAQ#streaming-json-parser
The last time I tried, I think the reason I gave up on JQ for large inputs was that the throughput would max out at 7mb/s whereas the same thing with spark SQL on the same hardware (MacBook) would max out at 250mb/s. So I started looking into using other solutions for big data while I use jq in parallel for small data in multiple files.
I will test it out again cause this was 4-5 years ago when I last tested it, but I believe jaq is still preferred for large inputs. Still I prefer for big data to use Spark/Polars/clickhouse etc.
- ripgrep PATTERN | xargs jq
- find PATTERN -exec jq
In both cases you have a large amount of data in your file system that is in JSON and you want to extract a subset of it for further processing. Ripgrep is an extremely fast way to do a content based search for a pattern and find is a fast way to do a file path based search. Then jq lets you extract the data.
For other types of data I use the same technique of first using ripgrep to find the candidate files and then piping to a text processor like awk/perl/ruby to actually process the data.
If you need FTS rather than regex search then SQLite FTS5 is my go to.
So a full example is something like:
`find . -name “*.pod.json” -print0 | xargs -0 -P 12 -I {} sh -c ‘jq -r “select(.spec.containers != null) | .spec.containers | to_entries[]” sh {} \; | jq -s ‘sort_by(.image)’`
Something like that so it’s sort of like a map-reduce you first narrow the subset of inputs by first finding by file pattern, then you pull out the relevant data from each in a parallel xargs, then you reduce it with a jq -s. This technique is used because jq is very slow on large files and your later processing scripts might be slow so the first step of a good pipeline is to first throw away all the data you don’t need first.