News for hackers. Stuff that matters.
Re:CR+LF Has a Long History by j1elo on Monday February 07, 2022 @11:00PM (#30254777)
What a coincidence to see CRLF discussed in HN front page, this bit me just yesterday!

While using the aws cli tool on Linux (the Docker containerized version) with the "text" output format, it turns out each output line ends with CRLF ("\r\n") instead of simply LF ("\n") as is common for any Linux cli tool, so I had to add a post-process step in my commands to remove CRs (a chained call to tr -d '\r'). Otherwise, my bash variables would end up containing those pesky carriage returns, and debug logs would show up unexpectedly bizarre.

In retrospect it's obvious what happened, but it got me confused for a good while. Why would it output CRLF at all? Also of course, not a single mention of this in any of the docs I was reading.

Re:CR+LF Has a Long History by thristian on Monday February 07, 2022 @11:53PM (#30255017)
The Unix convention is for "\n" alone to signify a line-break; when Unix programs talk amongst themselves (such as piping one to another), that's what they do.

The terminal convention (as described in the linked blog post) is for "\r\n" to signify a line-break. When a Unix program talks to a terminal, the kernel will (by default) automatically add or strip "\r" characters as needed.

Docker messes this up, since the tool inside the container is isolated from the environment where the "docker" command is run. Since it can't directly talk to the "docker" command's standard output, you have to manually choose whether it's connected to a pseudo-terminal ("docker run -t ...") or with a pipe ("docker run ...").

If the command runs with a pipe, no output translation will be done, but the program will likely be difficult to use interactively - there's likely to be no interactive prompts, no line-editing, etc.

If the command runs with a pseudo-terminal, then it will behave as if it's attached to a terminal - you're likely to get input prompts, ANSI colour codes, and "\r\n" line endings, even if the output of "docker run -t ..." is being piped to a file.

I don't know if this is the behaviour you hit, but I can easily imagine somebody running a tool in a docker container interactively with "docker run -it ..." until they got the behaviour they wanted, then just sticking that command in a shell pipeline like they would any other Unix tool, and getting a nasty surprise.

Re:CR+LF Has a Long History by j1elo on Tuesday February 08, 2022 @12:08AM (#30255097)
Thank you a lot for this explanation. I have a good understanding of the difference between using or not a pseudo-terminal (and its effects on Docker commands) but I had never learned before about this distinction on behavior of line breaks.

I'm launching the command with -it as suggested on https://docs.aws.amazon.com/cli/latest/userguide/install-cli...

However, now that you mention it, this page does comment on the usage of -it, so that might be it... thanks again for the hint!

Re:CR+LF Has a Long History by gnubison on Thursday February 10, 2022 @01:50PM (#30290710)
For the record, ONLRET is the relevant flag and you can disable it with “stty -onlret”. Then output newlines will stay newlines (which typically results in bizarre behavior if those raw newlines are sent to a terminal).
Re:CR+LF Has a Long History by CornCobs on Monday February 07, 2022 @11:34PM (#30254931)
The dos2unix tool is purpose-built for this!
Re:CR+LF Has a Long History by j1elo on Tuesday February 08, 2022 @12:00AM (#30255056)
You are right, that's the best tool to use for the job!

However, as it usually happens with scripts, adding non-coreutils dependencies becomes an issue (now the script needs to check for existence, request users to install if missing, etc...) For this use case which is so well scoped, a single character removal seems enough but in general I totally agree with dos2unix as the better choice.

Re:CR+LF Has a Long History by mananaysiempre on Monday February 07, 2022 @11:55PM (#30255024)
Note that dos2unix and unix2dos do what you expect when used as filters, but overwrite the specified file in place when given a file name. This is a footgun second only to one in pdftotext, arguably, which (unsurprisingly) refuses to work as a filter and (surprisingly) will silently create a matching .txt file when passed a single .pdf as an argument.
Re:CR+LF Has a Long History by spc476 on Tuesday February 08, 2022 @02:17AM (#30255746)
Why is that a footgun? If it just piped the output to stdout, then to convert a file, you then have to

    $ dos2unix file >x
    $ mv x file
In my mind, overwriting the file is the Right Thing.
Re:CR+LF Has a Long History by romwell on Tuesday February 08, 2022 @03:27AM (#30256081)
The Right Thing would be to require an explicit -o <output filename> option to avoid modifying files without getting user's consent first.

Especially because the changes it makes are irreversible (e. g. when a text file has mixed unix/dos line endings).

Re:CR+LF Has a Long History by Josteniok on Monday February 07, 2022 @11:45PM (#30254979)
Or flip (https://ccrma.stanford.edu/~craig/utility/flip/)! I've been carrying this thing around with me for years.
Re:CR+LF Has a Long History by bee_rider on Monday February 07, 2022 @11:11PM (#30254823)
Vague memory, so take it with a grain of salt, but I think this might be a way of getting the break to show up in an editor that expects \n or one that expects \r. Maybe most editors that expect one simply ignore the other?
Re:CR+LF Has a Long History by mdaniel on Tuesday February 08, 2022 @12:05AM (#30255084)
Related to that, the ansible "community.aws.aws_ssm" connection plugin[0] always returns the output of "raw:" tasks with "\r" appended to it, too, and I'd guess it's for a similar reason

0: https://github.com/ansible-collections/community.aws/blob/3....

Re:CR+LF Has a Long History by j1elo on Tuesday February 08, 2022 @04:39AM (#30256512)
UPDATE: The comment by user thristian [0] did indeed contain an explanation for this and hinted to what is the solution to this problem!

"docker run ..." should be scripted without "-t", in order to run the command without a pseudo-terminal, which then yields output with Linux style line breaks ("\n").

[0]: https://news.ycombinator.com/item?id=30255017

Re:CR+LF Has a Long History by kps on Tuesday February 08, 2022 @12:51PM (#30261780)
If for some reason you do want a terminal, you can run `stty -onlcr` to turn off the NL → CR LF conversion.
Re:CR+LF Has a Long History by ghshephard on Tuesday February 08, 2022 @04:01AM (#30256272)
Weird - I also had to write this exact thing:

   Creds ()
   {
       kubectl exec -ti $(getpod cred-api) -- printenv | grep --color=auto --color=auto RAWDATA | tr -d '\r'
   }
Re:CR+LF Has a Long History by j1elo on Tuesday February 08, 2022 @04:42AM (#30256532)
See my "UPDATE" comment; basically the issue seems to be that the output from "docker run" will contain DOS-type line breaks ("\r\n") if running with a pseudo-tty attached, so, with the "-t" argument. If that is not needed (e.g. when scripting) this argument can be dropped, and then the stdout comes clean with Linux style line breaks ("\n").

I don't know about kubectl, but if those "-ti" mean the same as in plain Docker, you could check if dropping the "-t" helps. After all, it looks like you are also scripting, so any kind of interactive input/output would probably be unexpected and undesired in your command.