kbrazil

There could be a major schema change that breaks the contract, but one of the nice things about JSON output is that it allows the creation of new fields without affecting downstream consumers.

That is, if I have a CLI program that spits out a list of IP addresses and one day I want to also output the corresponding dns names, I can simply add the "dns" field and existing pipelines will ignore the field and work just fine.

This is better than grep/awking/etc. unstructured text to STDOUT because, depending on how the author decides to add the new field, it can easily break existing pipelines that rely on the shape of the data to stay the same.

`jc` supports plugins:

https://github.com/kellyjonbrazil/jc#parser-plugins

`jc` author here. I've been maintaining `jc` for nearly four years now. Most of the maintenance is choosing which new parsers to include. Old parsers don't seem to have too many problems (see the Github issues) and bugs are typically just corner cases that can be quickly addressed along with added tests. In fact there is a plugin architecture that allows users to get a quick fix so they don't need to wait for the next release for the fix. In practice it has worked out pretty well.

Most of the commands are pretty old and do not change anymore. Many parsers are not even commands but standard filetypes (YAML, CSV, XML, INI, X509 certs, JWT, etc.) and string types (IP addresses, URLs, email addresses, datetimes, etc.) which don't change or use standard libraries to parse.

Additionally, I get a lot of support from the community. Many new parsers are written and maintained by others, which spreads the load and accelerates development.

Also, `jc` automatically selects the correct /proc/file parser so you can just do `jc /proc/meminfo` or `cat /proc/meminfo | jc --proc` without specifying the actual proc parser (though you can do that if you want)

Disclaimer: I'm the author of `jc`.

You can do it like this with Jello (I am the author):

    jello '[e.commit for e in _ if e.commit.author == "Tom Hudson"]'

Jello let’s you use python syntax with dot notation without the stdin/stdout/json.loads boilerplate.

https://github.com/kellyjonbrazil/jello

Interesting - the proc-pid-parser actually parses that file name just fine:

    $ echo '2001 (bad) S 1 2 3 4 5) S 1888 2001 1888 34816 2001 4202496 428 0 0 0 0 0 0 0 20 0 1 0 75513 115900416 297 18446744073709551615 4194304 5100612 140737020052256 140737020050904 140096699233308 0 65536 4 65538 18446744072034584486 0 0 17 0 0 0 0 0 0 7200240 7236240 35389440 140737020057179 140737020057223 140737020057223 140737020059606 0' | jc --proc-pid-stat

    {"pid":2001,"comm":"bad) S 1 2 3 4 5","state":"S","ppid":1888,"pgrp":2001,"session":1888,"tty_nr":34816,"tpg_id":2001,"flags":4202496,"minflt":428,"cminflt":0,"majflt":0,"cmajflt":0,"utime":0,"stime":0,"cutime":0,"cstime":0,"priority":20,"nice":0,"num_threads":1,"itrealvalue":0,"starttime":75513,"vsize":115900416,"rss":297,"rsslim":18446744073709551615,"startcode":4194304,"endcode":5100612,"startstack":140737020052256,"kstkeep":140737020050904,"kstkeip":140096699233308,"signal":0,"blocked":65536,"sigignore":4,"sigcatch":65538,"wchan":18446744072034584486,"nswap":0,"cnswap":0,"exit_signal":17,"processor":0,"rt_priority":0,"policy":0,"delayacct_blkio_ticks":0,"guest_time":0,"cguest_time":0,"start_data":7200240,"end_data":7236240,"start_brk":35389440,"arg_start":140737020057179,"arg_end":140737020057223,"env_start":140737020057223,"env_end":140737020059606,"exit_code":0,"state_pretty":"Sleeping in an interruptible wait"}

But the "magic" signature doesn't recognize it:

    $ echo '2001 (bad) S 1 2 3 4 5) S 1888 2001 1888 34816 2001 4202496 428 0 0 0 0 0 0 0 20 0 1 0 75513 115900416 297 18446744073709551615 4194304 5100612 140737020052256 140737020050904 140096699233308 0 65536 4 65538 18446744072034584486 0 0 17 0 0 0 0 0 0 7200240 7236240 35389440 140737020057179 140737020057223 140737020057223 140737020059606 0' | jc --proc             
    jc:  Error - Parser issue with proc:
                 ParseError: Proc file could not be identified.
                 ...

I can fix the "magic" signature (regex) to look for more parts of the stat file.

It doesn't look like jc suffers from this bug since the regex match is greedy:

    % echo '2001 (my (file) with) S 1888 2001 1888 34816 2001 4202496 428 0 0 0 0 0 0 0 20 0 1 0 75513 115900416 297 18446744073709551615 4194304 5100612 140737020052256 140737020050904 140096699233308 0 65536 4 65538 18446744072034584486 0 0 17 0 0 0 0 0 0 7200240 7236240 35389440 140737020057179 140737020057223 140737020057223 140737020059606 0' | jc --proc

    {"pid":2001,"comm":"my (file) with","state":"S","ppid":1888,"pgrp":2001,"session":1888,"tty_nr":34816,"tpg_id":2001,"flags":4202496,"minflt":428,"cminflt":0,"majflt":0,"cmajflt":0,"utime":0,"stime":0,"cutime":0,"cstime":0,"priority":20,"nice":0,"num_threads":1,"itrealvalue":0,"starttime":75513,"vsize":115900416,"rss":297,"rsslim":18446744073709551615,"startcode":4194304,"endcode":5100612,"startstack":140737020052256,"kstkeep":140737020050904,"kstkeip":140096699233308,"signal":0,"blocked":65536,"sigignore":4,"sigcatch":65538,"wchan":18446744072034584486,"nswap":0,"cnswap":0,"exit_signal":17,"processor":0,"rt_priority":0,"policy":0,"delayacct_blkio_ticks":0,"guest_time":0,"cguest_time":0,"start_data":7200240,"end_data":7236240,"start_brk":35389440,"arg_start":140737020057179,"arg_end":140737020057223,"env_start":140737020057223,"env_end":140737020059606,"exit_code":0,"state_pretty":"Sleeping in an interruptible wait"}

Edit: looks like I can tighten up the signature matching regex for the "magic" syntax per the issue found above. The greedy regex matching for the parser does seem to work fine, though.

jc[0] supports proc files. Converts them to JSON or YAML. (I am the author)

[0] https://kellyjonbrazil.github.io/jc/docs/parsers/proc

Fortunately `jc`[0] does parse `/proc/<pid>/stat` correctly. I, of course, originally implemented it the naive/incorrect way until a contributor fixed it. :)

    $ cat /proc/2001/stat | jc --proc
    {"pid":2001,"comm":"my program with\nsp","state":"S","ppid":1888,"pgrp":2001,"session":1888,"tty_nr":34816,"tpg_id":2001,"flags":4202496,"minflt":428,"cminflt":0,"majflt":0,"cmajflt":0,"utime":0,"stime":0,"cutime":0,"cstime":0,"priority":20,"nice":0,"num_threads":1,"itrealvalue":0,"starttime":75513,"vsize":115900416,"rss":297,"rsslim":18446744073709551615,"startcode":4194304,"endcode":5100612,"startstack":140737020052256,"kstkeep":140737020050904,"kstkeip":140096699233308,"signal":0,"blocked":65536,"sigignore":4,"sigcatch":65538,"wchan":18446744072034584486,"nswap":0,"cnswap":0,"exit_signal":17,"processor":0,"rt_priority":0,"policy":0,"delayacct_blkio_ticks":0,"guest_time":0,"cguest_time":0,"start_data":7200240,"end_data":7236240,"start_brk":35389440,"arg_start":140737020057179,"arg_end":140737020057223,"env_start":140737020057223,"env_end":140737020059606,"exit_code":0,"state_pretty":"Sleeping in an interruptible wait"}

[0] https://kellyjonbrazil.github.io/jc/docs/parsers/proc_pid_st...

Both ways work:

    $ jc ifconfig lo0

    $ ifconfig lo0 | jc --ifconfig

Good news - `jc` supports JSONLines output for many commands and file-types, too!

Also note that you are looking at plaintext output here. By default `jc` and other JSON filtering tools do syntax highlighting when outputting to the terminal so it's actually quite easy to read JSON these days.

Hi - jc author here. Just a couple points:

The casting to int/float is not done unless the underlying values are predictably documented to be numbers. There are rare cases where auto int/float conversions are done, but:

1) This is always documented, and

2) You can turn this functionality off via the —raw flag

Also, predictable schemas are documented with each parser. (e.g `jc -h —-arp`)

Feel free to open an issues on GitHub if you would like to recommend a parsers! Also, the latest version of jc supports parsing of /proc files, which may also help.

Hi there! `jc` author here. Yes, it is a documented caveat[0] that the `C` or `en_US.UTF-8` locales should be used for best results.

It's not unheard of for tools to require `C` locale for proper parsing:

    $ LC_ALL=C ls -l | jc --ls

This is one of many inherent issues with using unstructured text as an API. That's why I believe there should be a JSON (or at least some other widely used format[1]) option for tools that have output that would be useful in scripts.

[0] https://github.com/kellyjonbrazil/jc#locale

[1] formats should have good library support across many languages and nice filter/query capabilities from the command-line

You can also run `jc -h --dig` to get the parser details that include the schema.

Having true JSON Schema[0] is being considered, but on the back-burner due to the sheer number of parsers to build schemas for. Also, it is more difficult to accurately define the schema for a small subset of parsers since their command output are so variable.

[0] https://json-schema.org/

`jc` is available today as a compiled binary with all dependencies self-contained. The binaries and OS package installers are available under the GitHub Releases:

https://github.com/kellyjonbrazil/jc/releases

This is still python under the hood and not as small of a binary as I would like, but it does work.

Hi there - author of `jc` here. I originally intended to have auto-detection but put that on the backburner to focus on creating parsers, especially after introducing the magic syntax.

I did implement auto-detection for `/proc` file parsers so you can just do:

    $ cat /proc/foo | jc --proc

or

    $ jc /proc/foo

But you can specify each procfile parser directly if you want to as well.

Hi there - `jc` author here. `jc` can also output in YAML format with the `-y` flag. It is fairly trivial to add other options in the future since `jc` just turns the text into objects which can be serialized to many different formats.

For example:

    % jc -y date
    ---
    year: 2022
    month: Nov
    month_num: 11
    day: 3
    weekday: Thu
    weekday_num: 4
    hour: 9
    hour_24: 9
    minute: 0
    second: 22
    period: AM
    timezone: PDT
    utc_offset:
    day_of_year: 307
    week_of_year: 44
    iso: '2022-11-03T09:00:22'
    epoch: 1667491222
    epoch_utc:
    timezone_aware: false

Hi there - I'm the author of `jc`. I also created `jello`[0], which works just like `jq` but uses python syntax. I find `jq` is great for many things but sometimes more complex operations are easier for me to grok in python.

[0] https://github.com/kellyjonbrazil/jello

Similar in concept to jello[0] which works like jq but uses python syntax.

[0] https://github.com/kellyjonbrazil/jello

You could do something like this in pure python without the json loading boilerplate with jello[0]. An interactive TUI for jello called jellex[1} is also available. (I am the author)

[0] https://github.com/kellyjonbrazil/jello

[1] https://github.com/kellyjonbrazil/jellex

For those who are comfortable with python, I created Jello[0], which works like jq but uses python syntax.

I also created Jellex[1], which is a TUI built on Jello to assist with building the python queries.

Jello gives you the power of python but without all of the boilerplate, so it’s nicer to use in Bash scripts.

[0] https://github.com/kellyjonbrazil/jello

[1] https://github.com/kellyjonbrazil/jellex

What I was getting at is:

    cpu: 0.2
    kw_read_s: 0.12

is describing a single 'object'. How do you describe an array or list of these objects? Something like:

    cpu: 0.2
    kw_read_s: 0.12

    cpu: 1.3
    kw_read_s: 0.4

Do you use a blank line to denote a new 'object'? In JSON it would be done this way, and this conforms to how the vast majority of command output maps (as they tend to be rows of columnar data):

    [
      {
        "cpu": 0.2,
        "kw_read_s": 0.12
      },
      {
        "cpu": 1.3,
        "kw_read_s": 0.4
      }
    ]

The other benefit to JSON here is that the formatting doesn't matter. This could also be expressed as a block of text with no spaces or newlines between elements:

    [{"cpu": 0.2,"kw_read_s": 0.12},{"cpu": 1.3,"kw_read_s": 0.4}]

Finally, this can also be streamed, using JSON Lines:

    {"cpu": 0.2, "kw_read_s": 0.12}
    {"cpu": 1.3, "kw_read_s": 0.4}

New parser contributions for JC are always welcome!

Maybe not, but more often than not you really have an array of objects, and that's where JSON helps.

Yep, in JC you can see the schema for any parser like so:

$ jc -h --dig

If you like python you might check out jello[0]. I basically wrote it to give you the power and simplicity of python without the boiler plate in a jq-like form-factor. Jello also allows you to use dot-notation instead of dict bracket notation, so it does make things easier on the command line.

Also, there is jellex[1], which is a TUI wrapper around jello that can help you build your queries.

[0] https://github.com/kellyjonbrazil/jello [1] https://github.com/kellyjonbrazil/jellex

Grep would not be able to do numeric comparison, would it?

That’s why I created jello[0]. You get the power of python without the boilerplate so the experience is closer to jq with python syntax.

[0] https://github.com/kellyjonbrazil/jello