News for hackers. Stuff that matters.
Re:The CSV Virtual Table by hnlmorg on Wednesday March 09, 2022 @04:28AM (#30612550)
That's a fair question. The shell has been around for > 8 years (and been my primary shell for around 5 of those years) so it made sense to integrate it into the shell because I could then leverage the existing code that transparently handles different data formats. That shell also sends metadata about the documents being piped which means you effectively have typed pipes (like Powershell but backwards compatible with POSIX tools). So it means you can do more with the table afterwards too.

And since the shell is already good for handling structured data (like `jq` et al but not limited to JSON) it also made sense to allow inlining of SQL for when you're working with relational data.

An example of this last point is a recent problem I had at work where I wanted to see which users had 2FA enabled. Unfortunately this service provided a user list API (returns a complex JSON document) and a separate user meta data API (again JSON). The meta data didn't have names on it and the user list didn't have 2FA details on it. So I had two JSON documents that weren't tables but ostensibly held relational data. Using the shell I could convert the two JSON documents into two tables and then use the SQL inlining `select` command to do the relational query. The whole thing was 3 lines of code whereas to do the equivalent in Python / Typescript / etc would have been much more verbose.

Re:The CSV Virtual Table by dotancohen on Wednesday March 09, 2022 @04:39AM (#30612628)

  > That shell also sends metadata about the documents being piped which means
  > you effectively have typed pipes (like Powershell but backwards compatible
  > with POSIX tools). So it means you can do more with the table afterwards too.
It looks like such a shell would greatly benefit from a stdmeta file descriptor, as described here:

https://unix.stackexchange.com/questions/197809/propose-addi...

The idea is that such a file descriptor would output lines that are not part of the data yet are not errors. One example usage would be to output the headers of commands such as ps, and murex could use such stdmeta lines as column names.

Re:The CSV Virtual Table by hnlmorg on Wednesday March 09, 2022 @04:46AM (#30612685)
That would be awesome but I think what we need is something a little more programmatic. eg a standard way of passing Content-Type / MIMEs with fields for meta-data. A hypothetical example might look like this:

  {
    "Content-Type": "application/table",
    "Stdin": "ls",
    "Meta": {
      "Columns": [ "filename", "date", "permissions", "owner" ]
    }
  }
I'm not suggesting that data should be JSON formatted though. Nor even have fields identical to the above. But a standard agreed base so that applications can understand what the pipe is without the developers having to write a thousand different case statements for each different stdmeta content but while still allowing some enhanced functionality for common commands.

In an idea world this would be an API rather than file descriptor but that would be backwards incompatible and even the FD use case feels like nice thing that is always going to be out of reach. :(

Thanks for sharing that by the way. I hadn't seen that post before.