News for hackers. Stuff that matters.
Re:If you use GNU grep on text files, use the -a (--text) op... by gpvos on Tuesday April 21, 2020 @04:40PM (#22939087)
Why is it that in recent years, with this and the more recent ls quoting fiasco, maintainers of longstanding UNIX utilities suddenly got the urge to fix what isn't broken?
Re:If you use GNU grep on text files, use the -a (--text) op... by chubot on Tuesday April 21, 2020 @05:45PM (#22939652)
What's the 'ls quoting fiasco'?

Actually I recently found that coreutils and ls behave fairly well with funny filenames:

Here is an invalid utf-8 byte and then a valid utf-8 sequence

    $ x=$'\xce\xce\xbc'
    $ touch "$x"
You can list it:

    $ ls
    ?μ
And here 'ls' does better than other tools that display filenames. It shows the invalid byte and then keeps decoding with error recovery:

    $ ls --escape
    \316μ
However GNU stat (which I think is also in coreutils) does something similar, but weirdly messed up:

    $ stat *
    File: ''$'\316''μ'
(it looks like it's outputting a valid shell string, except with extra quotes)

-----

Most command line tools are not aware of stuff like this. For example you can touch "x$ANSI_TERMINAL_CODES" and if you do "bash x??" or "python x??", then your terminal will change color because of the escape codes printed back to the terminal.

I just changed Oil to use a well-defined format I called QSN (quoted string notation):

http://www.oilshell.org/blog/2020/04/release-0.8.pre4.html#t...

It adapts Rust's string literal syntax to express arbitrary byte strings precisely and losslessly. (JSON can't express arbitrary byte strings.)

The QSN encoder does UTF-8 decoding with a specific error recovery mechanism. So it's basically like what ls and stat do, but it's more precise.

(If anyone is interested in QSN, please contact me. I think it's more generally useful in a lot of places. It's something we already do but it's precise like JSON.)

Re:If you use GNU grep on text files, use the -a (--text) op... by _jal on Tuesday April 21, 2020 @06:28PM (#22940016)
They broke it in 2016.

https://www.gnu.org/software/coreutils/quotes.html

At least with Gnu, you can recompile your own, non-broken version, which is the only saving grace of these stupid, trendy changes.

Re:If you use GNU grep on text files, use the -a (--text) op... by Anthony-G on Tuesday April 21, 2020 @06:46PM (#22940132)
Not broken at all. I really don’t understand the hate for this change.

I deal with a lot of filenames with spaces and think this change is a great improvement for listing such files. With this change it’s much easier to see where one filename ends and the other begins. Before this change, I had to use the `-1` option to ensure that each filename was listed on a line by itself. Now the filename listings are much more readable and it takes less cognitive effort to take it all in.

The way it handles filenames with ASCII apostrophes/single quotes works particularly well (wraps the filename in double quotes instead of single quotes) and makes it very easy to copy and paste filenames to and from the terminal.

Best of all, this change only applies when standard output is a TTY device so this does not break any shell scripts (even though parsing `ls` is a bad idea in any case) and is still compliant with the POSIX specification[1] which states that “If the output is to a terminal, the format is implementation-defined”.

1. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/l...

Re:If you use GNU grep on text files, use the -a (--text) op... by chubot on Tuesday April 21, 2020 @09:08PM (#22941016)
The old behavior was the bad one; the new behavior is good. And that link explains why very well.

I guess I should have figured that oblique references to "ls quoting fiasco" is shorthand for "I don't understand what's wrong but I'm angry about it..."

(On the other hand I would say the grep -a issue is bad both before and after because either way it relies on autodetection. The fundamental issue there is that there is too much variance in encodings, which isn't easy to fix. Luckily UTF-8 is growing in popularity, and it doesn't have this issue because it doesn't require metadata for extremely common operations like "find ascii substring".)

Re:If you use GNU grep on text files, use the -a (--text) op... by wruza on Tuesday April 21, 2020 @11:50PM (#22941924)
”The old behavior was the bad one; the new behavior is good”

If you’re a human, yes. If you’re a script, it breaks you in half. If you’re a script that has to run on various versions, then maybe it’s time fix yourself and use find. You’re a sophisticated script after all, not one of these who require a human with a debugger. Modern culture may not appreciate that little ‘compat’ thing, but it is essential if you want something to continue to work and not just stop and wait for someone’s educated guesses. Good software doesn’t point fingers at you, it just works. I remember how recently I wanted to check network interfaces on some machine and commanded ‘ifconfig’. Now it’s called ‘ip a’, and there is no ifconfig. I can guess the reason – ifconfig was bad and ip is good. There is also an eternal “FAT” label issue in unetbootin app, which resurrects every time Apple changes its fdisk output format (in every release, as it seems). The workaround is to run it with a cli option – a very thing that unetbootin was created for to skip. This is what makes systems so much fun. Without all these cool things, we would just sit there and cry over our uselessness.

ed: I read below that ls does that in interactive mode only, maybe it’s not that bad then.

Re:If you use GNU grep on text files, use the -a (--text) op... by int_19h on Wednesday April 22, 2020 @04:21AM (#22943359)
You can't just expect compatibility about random things - that's why we have formal contracts: standards, specifications, documentation. A human-readable output of any app, in particular, should never be assumed to be stable, or to have a specific format (even if observations imply it), unless its docs specifically say otherwise.
Re:If you use GNU grep on text files, use the -a (--text) op... by foldr on Wednesday April 22, 2020 @03:28AM (#22943109)
A script will see the old-style output. The new-style output is tty only
Re:If you use GNU grep on text files, use the -a (--text) op... by steerablesafe on Wednesday April 22, 2020 @03:17AM (#22943050)
Hats off if your script used ls and handled weird characters in filenames just fine.
Re:If you use GNU grep on text files, use the -a (--text) op... by _jal on Tuesday April 21, 2020 @10:05PM (#22941289)
> but I'm angry about it.

Take it to Reddit. I know perfectly well what's going on, and also know better then to discuss it with people who act like that.

Re:If you use GNU grep on text files, use the -a (--text) op... by Delk on Tuesday April 21, 2020 @07:54PM (#22940584)
It would seem that calling ls with the -N switch would disable the quoting. Might be easier to just have ls aliased to "ls -N" rather than recompiling. Unless the -N switch also does something else you don't like.
Re:If you use GNU grep on text files, use the -a (--text) op... by arbitrage on Wednesday April 22, 2020 @11:28AM (#22946576)
again, aliasing it is fine for interactive use.

the point is that this breaks an unbelievable amount of already deployed scripts. The new functionality should be optional, and accessed via switches and aliases if you like it.

this was very much a change only a few people liked, that they decided to force down literally everyone else's throats. it's very poor stewardship.

Re:If you use GNU grep on text files, use the -a (--text) op... by Delk on Wednesday April 22, 2020 @04:43PM (#22949991)
Okay. The GNU page says the quoting is only done when the output is a terminal, so it shouldn't generally affect scripts. Although considering that the world of unix shells is fairly complex, I wouldn't be surprised if there was some kind of a weird but explainable situation where some kind of a script would still break, so I suppose it could be a problem somewhere.
Re:If you use GNU grep on text files, use the -a (--text) op... by boring_twenties on Tuesday April 21, 2020 @06:03PM (#22939837)
GNU ls prints the filenames already-quoted as of late:

    % touch foo
    % touch 'bar baz'
    % ls
    'bar baz'   foo
Re:If you use GNU grep on text files, use the -a (--text) op... by knolax on Tuesday April 21, 2020 @06:01PM (#22939815)
A google search for 'ls quoting fiasco' shows this thread as the top result. Could you explain what it is?
Re:If you use GNU grep on text files, use the -a (--text) op... by smegma2 on Tuesday April 21, 2020 @10:13PM (#22941331)
There was a change in ls which caused it to quote file names with spaces or special characters instead of (I think?) escaping them. Some people were upset because they considered this a breaking change. I'll just add that ls only does this if it's running interactively so I wouldn't expect it to break scripts.
Re:If you use GNU grep on text files, use the -a (--text) op... by ansible on Tuesday April 21, 2020 @10:46PM (#22941534)
I'm a little surprised people are using ls in scripts. I'm normally using find, with the mill separator to deal with spaces in filenames.
Re:If you use GNU grep on text files, use the -a (--text) op... by ansible on Wednesday April 22, 2020 @07:23AM (#22944389)
... null separator ...

I thought I proof-read that sentence when I wrote it...

Re:If you use GNU grep on text files, use the -a (--text) op... by lazyjones on Wednesday April 22, 2020 @12:45AM (#22942285)
Because of the prevalence of UTF-8 and its consequences/pitfalls?
Re:If you use GNU grep on text files, use the -a (--text) op... by gpvos on Wednesday April 22, 2020 @03:40AM (#22943173)
That's the best explanation I can think of too.
Re:If you use GNU grep on text files, use the -a (--text) op... by TheGoddessInari on Wednesday April 22, 2020 @04:26AM (#22943389)
"GNU's Not Unix!"