See https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1...
> 3.387 Text File
> A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character.
So, if you have some non-printable characters like BEL/␇/ASCII 0x07, that's still a text file.
(and I believe what bytes count as a valid character depend on your `LC_CTYPE`).
But the moment you have a line longer than {LINE_MAX} bytes (which can depend on which POSIX environment you have), suddenly your text file is now a binary file.
The definition should read "one or more lines" instead or (probably better) specify that a text file contains "zero or more characters".
> 3.185 Line
> A sequence of zero or more non-<newline> characters plus a terminating <newline> character.
So a file with some characters but no trailing newline is reported by `wc -l` as having zero lines.
Please don't use that underdefined joke of a spec. Define "PosixJson" and use that instead. Right now it's not even clear what the result of parsing {"a": 1234678901234567890} is. Is this a parse error? A bigint? A float/double? Quiet wraparound? Something else? I've seen all these behaviors in real world JSON implementations across different languages.
But then, how well would it work for ad-hoc usage, which is probably one of the biggest uses of shells?
You never have to worry about whether you're dealing with ASCII vs. UTF-8, but rather if you're dealing with UTF-8 vs. ISO-8859-1, or worse, Shift JIS or similar.
% java -Dfile.encoding=UTF-32 Test | hexdump -C
00000000 00 00 00 48 00 00 00 65 00 00 00 6c 00 00 00 6c |...H...e...l...l|
00000010 00 00 00 6f 00 00 00 2c 00 00 00 20 00 00 00 77 |...o...,... ...w|
00000020 00 00 00 6f 00 00 00 72 00 00 00 6c 00 00 00 64 |...o...r...l...d|
00000030 00 00 00 0a |....|
00000034
From quick googling it seems that glibc does not support it, so it should not happen.`iconv` does, and this is enough in common. Among with tons of eerie EBCDIC/whatever...
I regularly use `iconv -t utf-32be | hd` to look what a bizarre sequence is denoting yet another weird symbol like an itchy hedgehog.
And what is a real reason to disallow this?
Command line scripting is supposed to be adhoc and hack.
If you can't get the immensity of the cleverness of Unix foundations, you should not talk about them.
That idea is what made it possible for you to type that sentence in the first place.
First: text is not well defined. Is it ASCII? Is it UTF-8? Some programs can spew UTF-32 with proper locale configured, it's a mess.
Second: encoding and decoding of objects to text is not defined at all. Those problems with filenames is just one example. Using newline as a separator is a natural thing that is easy to implement, yet it is wrong.
In my opinion two things should be done:
1. Standardise on UTF-8. No other encodings allowed.
2. Standardise on JSON. It is good enough to serve as universal exchange format, tools like `jq` exist for some time now.
So any utility must read and write JSON objects with some standard env set. And shells can be developed with better syntax to deal with JSON. This way you can write something like
`ps aux | while read row; do echo ${row.user} ${row.pid}; done`