Re:Q: A faster re-implementaiton of jq written in Reason Nat...
by benibela on Monday September 14, 2020 @09:59AM
(#24470209)
files might be faster, because you can mmap them?
[ Parent
| on HN ]
Re:Q: A faster re-implementaiton of jq written in Reason Nat...
by f311a on Monday September 14, 2020 @10:09AM
(#24470326)
Most people work on compressed JSON lines files. Sometimes they are stored on s3. Files do not give flexibility.
When using jq, I can do a lot of things:
aws s3 cp s3://bucket/file.json.gz - | zcat | head | jq .field | sort
Re:Q: A faster re-implementaiton of jq written in Reason Nat...
by jerf on Monday September 14, 2020 @10:23AM
(#24470492)
Another common thing you can do is accept a generic stream as input, but have some code that penetrates the abstraction a bit to see what kind of stream it is, and if it is a file or something, do something special with it to go even faster. This way, you start with something maximally useful up front, and easy to use, but you can optimize things based on details as you go.
That's how Go's static file web server works. It serves streams, but if you happen to io.Copy to that stream with something that is also an ∗os.File on Linux, it can use the sendfile call in the kernel instead. (A downside of making it so transparent is that if you wrap that stream with something you may not realize that you've wrecked the optimization because it no longer unwraps to an ∗os.File but whatever your wrapper is, but, well, nothing's perfect.)
Re:Q: A faster re-implementaiton of jq written in Reason Nat...
by Spivak on Monday September 14, 2020 @10:24AM
(#24470501)
Which is sorta-kinda the same thing as
jq .field <(aws s3 cp s3://bucket/file.json.gz - | zcat | head) | sort
which is more annoying to type but works.
Re:Q: A faster re-implementaiton of jq written in Reason Nat...
by dullgiulio on Monday September 14, 2020 @12:37PM
(#24472149)
Redirection is not a parameter, meaning that would still not work with this Q tool.
Re:Q: A faster re-implementaiton of jq written in Reason Nat...
by jraph on Monday September 14, 2020 @01:07PM
(#24472515)
This is not a simple redirection. cmd <(subcmd) is a bashism that redirects the output of command subcmd to something that looks like a regular file to command cmd. Command cmd receive a path at the place the <(subcmd) syntax is used. Different from cmd < f, which redirects the contents of file f to cmd's input.
So, this should work :-)