I feel like `fq` has a query path language that's kind of generic across lots of file types. It can be fairly verbose for that reason. I was using it to debug MsgPack documents and it was a lot less intuitive than just using some dotted string paths with `jq`.

https://github.com/wader/fq/

Hey, fq author here. Happy to hear it's useful! could you elaborate a bit more how it was less intuitive? fq's query language is jq with some small additions so i wonder if you might mean the decoded structure is more detailed/verbose as it includes all the "low level" details? maybe your looking for the "torepr" function that converts the detailed structure into the "represented" value?

Never seen this tool before but it looks quite handy. Thanks!

hope it can be of use!

Yes that's exactly what I meant, the MsgPack documents had quite a detailed structure.

torepr didn't quite work for me as I was dealing with objects containing large binary blobs and it was awkward.

fq is a great tool and I shouldn't have suggested this was a problem unique to it! I think this kind of "issue" is inevitable when dealing with so many types of input. And to be honest I struggle hard using jq as well for anything other than very basic paths, due to infrequent usage.

I see, thanks for replying and no worries! yeap some of the "self-describing" formats like msgpack, cbor etc will because of how fq works have to be decoded into something more of a meta-msgpack etc.

About blobs, if you want to change how (possibly large) binaries are represented as JSON you can use the bits_format options, see https://github.com/wader/fq/blob/master/doc/usage.md#options, so fq -o bits_format=md5 torepr ...

I can highly recommend to learn jq, it's what makes fq really useful, and as a bonus you will learn jq in general! :)

Oh that's a great tip, I will try to remember this next time I'm touching the msgpacky parts of my app!

'Nested maps and arrays of strings, numbers and booleans'? I propose the name AMofBNS :)

Is it anything else than a tree with properties for each node?

I think you could well apply jsonpath to yaml, except for the different data types, which is what makes you need xmlpath, jsonpath, file path, css and so on. If you're willing to do some automagic conversions, you could probably do that right now, if you write the code for it.

A tree with properties indeed.

They vary in syntax and how they deal with scalars, objects and collections. Having to write 'myKey' is a bit unfortunate in json for example. Still, for any document larger than 20 lines yaml will fall apart easily.

Xml (dialects) mark the beginning and end of nodes explicitly, which deviates from json/yaml. Xml can represent nodes within the value, which is impossible for json/yaml. To convert such xml value to the latter, you have to break up such a node in fragments and represent them as a collection in json/yaml.

> Xml can represent nodes within the value, which is impossible for json/yaml. To convert such xml value to the latter, you have to break up such a node in fragments and represent them as a collection in json/yaml.

Are you talking about XML like `<text>Something something <para> inside </para> something else</text>`? I thought this would also presented as the text element having three children, the text "Something something", the <para> tag with its subtree, and the text "something else". Am I misremembering?

Exactly, that is the «breaking apart» I was talking about. Xml is the more natural way for humans to express that particular use case.

But my point is that an XML parser would still present that as three children, right?

It's just when writing the document that you get the advantage, the data model is the same.

Jup, it is human ergonomics, but a mismatch for your average programming data structure.

I think that is one reason why json took of for exchanging machine readable data, xml is too expressive, and leans towards document authoring.

Agreed, a lot of the features that make XML very nice for embedding tags in text documents make it very inefficient at expressing explicit tree data structures.

I doubt that, if I had my money on it I’d guess the main reason is var obj=JSON.parse(string) and its ease.

That is half the reason. JSON was limited to Javascript when SOAP/WSDL was king for data exchange. If I remember correctly, XML apis predates json in Javascript. IE5 and XML were really hot back then. xml.loadString() is equally simple.

Json got introduced because XMLHttpRequest was an IE-only component. Data exchange was done mostly between servers, back then the front-end was really dumb. Why xml got slowly replaced is because that for machine generated data, json proved to be sufficient. Any json library is easier to work with then their xml counterpart, exactly because of what we discuss here.

The complicated part is, that this tree has different types of nodes. Some support properties and child's, some not. So I would call it a chimera tree or mixed tree, or something like that.

For a query language, you can generalize to all nodes supporting all things. You will be able to write some queries that will never return results, at least in a given input data format. That does not seem so bad.

No it makes so much more sense to introduce nil and then constantly pepper your code with ? (Null-colasecing) Operators everywhere

(Looking at you, jq, though to be fair JSON itself supports null...)

Coincidentally, this is exactly what XPath and XQuery do. E.g. text nodes don't have children nor attributes, yet it is perfectly legal to do text()/foo or text()/@foo - and the result is an empty sequence.

If you want this kind of semantics with JSON, take a look at https://www.jsoniq.org

You could say it is 'object-oriented' since the ON stands for Object Notation.

nested records and arrays?

As to cross format path querying, I see limited value in such an endeavour; the reason that one deals with such formats is generally because it is used as an interface to configure some tool, but moving across different tools, besides the path queries, even the configuration schemas differ, and having a cross format converter would do you no good. The fact that many such configuration formats support subtly (or drastically) divergent types of objects does this idea no good either.

Speaking of this topic, I would be very interested in a CLI tool that translated between different dialects of regex. As opposed to general data or configuration, I find the case for cross format conversion very compelling for regexes; the objects that different regex languages deal with are functionally completely identical. I would be very happy to be able to craft a regex to search for something in vim, then convert the regex to grep or use another tool on the shell, then perhaps adapt it to some other scripting languages e.g. Javascript/Ruby/Perl.

Indeed I am sad that I am not finding any CLI tools (non web-based) for this beyond https://github.com/Anadian/regex-translator . Am hopeful for more suggestions.

I think one popular name, independent of the format, is to call them documents. Or more specifically, the type of database which use them often runs under that term. Maybe specify it as data-document to not confuse it with freeform office-documents.

But one problem is, that each format has slightly different ways it works. Some have nodes which support properties, some not. This makes building a proper query-language a bit more complicated if it should not end up ugly.

What's the problem with nodes supporting or not some properties? For querying you can treat all nodes as supporting all things, and the user just needs to write a query that acutually works, which is what they would need to do anyway.

Yes, if you want to make the query language reject as incorrect queries that specify a property on a node that can't have properties, it is messy. But is this really necessary? You can write faulty queries anyway. And it's not a programming language for complex systems, where it really helps to prevent mistakes early. SQL works fine with no type safety and so on.

The problem is that you can't easily reuse a query with different formats, with a swallow designed language. You will still end up writing format-targeted queries, which then open the question of why you would even bother in the first place using a watered down language, instead of a format-optimized one.

Another issue is that the term "document" is also used to refer to XML or HTML.

XML is for data-exchange, competing with JSON and others. So I have no problem putting it to the data-documents, even though it's more a frankenstein. But HTML is an office-document, used for freetext, nobody really should use it for data, even though sometimes it's used that way.

XML is also used, literally, for office documents (Word, Excel, etc).

So are JSON and YAML. The point is whether you have a programmatic and structured handling of a document's content, or whether it's random, where every line and word can demand for a different parser.

Tree-structured data

The general name is semi-structured data, as opposed to structured data (tables) and unstructured data (free text).

The Readings in Database Systems book calls it "general purpose hierarchical data format" at some point, which seems the most fitting name for me.

You generally don't see yaml or XML be called that, but there's some information on the net that you can find about it.

Structually, it's a tree. JSON cannot include circularity (like JavaScript objects can) so it's either an object, or nested objects.

grove - tree decorated w/ key-value pairs. eg JSON, XML w/o text nodes

object graph - grove plus object references. eg serialization formats

knowledge graph - object graph where parent-child relations are explicit, using subject-verb-object clauses. like how Java Spring "flattens" object graphs.

document - grove plus random text nodes

Although there are similar data structures, not all of them are supersets nor subsets of the data structures of JSON. For example, some data types that JSON lacks are:

- Integers (including 64-bit integers and longer)

- Non-finite floating point (Infinity, NaN)

- Keys of types other than strings

- Non-Unicode strings (e.g. byte sequences, TRON code, etc)

- Date/time

- Links

Additionally, they may differ of whether or not the order of keys should be retained.