Yep, some friends of mine did this with JSON, but didn't make the schema explicit like I mean to: https://github.com/benbernard/RecordStream
> Better yet, unify the shell with a virtual machine that is used to implement the OS, and have everything available as 1st class Objects.
Please no. This is the Microsoft PowerShell approach, where everything is a .NET object. Once you start dictating representations of objects, you are dictating far too much about the implementation of individual pipeline nodes.
Yes, you have to agree on a serialization format (JSON, Protocol Buffers, etc), but that's not the same thing. From a serialization format you can represent the data however you see fit in your process. For example, a C++ user might represent a string as a std::string object whereas a Python user would represent it as a native Python string.
The VM-based approach (like PowerShell) defines an in-memory tree representation, namely .NET objects. This means that you can't really interoperate with this stack unless you use .NET too, since you don't have an easy way of converting .NET objects to your own objects.
To be super concrete, I can't read a file containing protos without knowing their type, what fields they contain, etc. I can however read JSON just fine without knowing the precise schema being encoded.
ORS=: OFS=, ls -l # rw-r--r--:1:uname:gname:1025:Jun 1:somefile.txt,...,...
Add a specifier for dates, usernames, groupnames, etc. DSF="%Y-%m-%d" ls -l
Until you need to incrementally process 2GB of data.
There are a variety of serialization schemes that are quite easy to parse and would be suitable for the output of most Unix command-line tools. BEncode from bittorrent would do nicely, and it's quite easy to parse. JSON would do nicely as well.
Better yet, unify the shell with a virtual machine that is used to implement the OS, and have everything available as 1st class Objects.