News for hackers. Stuff that matters.
Re:What's New in POSIX 2024 by enriquto on Tuesday October 29, 2024 @03:27AM (#41980339)
> We’ve established that, yes, pathnames can include newlines. We have not established why they can do that. After some deliberation, the Austin Group could not find a single use-case for newlines in pathnames besides breaking naive scripts. Wouldn’t it be nice if the naive scripts were just correct now? Ok, that might be a bit much all at once. We’re heading there though!

Oh my god. This makes me so happy. This is the most lovely think I've read in the world of computing since the unix gods decided that newlines were to be a single character.

The philosophy underlying the sentence "Wouldn’t it be nice if the naive scripts were just correct now?" is incredibly positive. We are surrounded by arrogant jerks who break old code by aggressively enforcing stricter compliance of some stupid rules. But here come these posix heros who do the exact opposite: make old code correct! There is hope in mankind after all.

Re:What's New in POSIX 2024 by anal_reactor on Tuesday October 29, 2024 @03:41AM (#41980425)
It's a bandaid on a wider problem: the design of Unix shell is bonkers and the whole thing should be deleted. Why? Because I haven't seen any other tool ever have so many pitfalls. Take n random languages and m random developers and tell them to loop over a string array and print its contents, and count how many correct programs you get on average per language. There will be easy languages, then difficult languages, then a huge gap, then Unix shell because in your random sample you managed to get one guy who has PhD in bash.
Re:What's New in POSIX 2024 by enriquto on Tuesday October 29, 2024 @04:19AM (#41980828)
> loop over a string array

Dear anal_reactor, what is a "string array"? I have used unix shells since nearly 30 years and never heard about them. And I consider myself a script-fu master!

There are two array-like constructions in the shell: list of words (separated by spaces) and list of lines (separated by newlines). Both cases are implemented as a single string, and the shell makes it trivial to iterate through its components.

Re:What's New in POSIX 2024 by ManBeardPc on Tuesday October 29, 2024 @04:30AM (#41980902)
That is exactly the problem many people have with it. Encoding „arrays“ this way is foreign to everyone who comes from „normal“ programming languages. Both variants lead to problems because either character can occur in elements, worst case scenario they contain both at the same time. I can see why this leads to confusion and bugs.
Re:What's New in POSIX 2024 by skydhash on Tuesday October 29, 2024 @08:06AM (#41983025)
It’s like people saying they won’t learn French because it has a different grammatical structure. There’s no “normal” natural language. If you’re used to the C-like syntax, learning C-like language will be easy. But that’s not an argument to say Lisp is confusing.
Re:What's New in POSIX 2024 by ManBeardPc on Tuesday October 29, 2024 @08:52AM (#41983346)
That's why I put normal in quotes. There is however more to it than having a different grammatical structure: It works different from many commonly used languages that have actual arrays/lists where elements can contain anything the type allows. If you come from any of the common modern programming languages (lets say Java, Kotlin, C#, JS/TS, Python, Swift, Go, Rust, etc.) and expect something similar (because many of them are very similar) you will be confused. Using spaces or newlines to encode elements in a single string is just not robust and leads to easy to make mistakes.
Re:What's New in POSIX 2024 by skydhash on Tuesday October 29, 2024 @09:38AM (#41983725)
Most of these languages were created long after bash and the other shells. The fact is that shell scripts allows for unquoted strings and quoting is a specific operation, not syntax. Also shell scripts were meant for automations, not for writing general programs. The basic units are commands, arguments, input, output, files,… so the design makes these easy to manipulate.

I’m not saying that we can’t improve, but I’m more in favor of making the tool more apt to solve a problem than making it easier to learn. Because the latter often wants to forego the requirement of understanding the problem space.

Re:What's New in POSIX 2024 by ManBeardPc on Tuesday October 29, 2024 @12:22PM (#41985958)
Yes, these are newer. I mainly wanted to make the point that it is confusing if you are new to bash and come from these newer languages with the wrong expectations. The concise nature and many subtle details makes it very difficult for beginners and infrequent users.

Compare this to the newer programming languages where you explicitly call something with speaking names like .Trim(), .EndsWith(), support from compiler and IDE.

In my experience automation and general programs often are the same thing once things get more complicated. Bash scripts usually grow rapidly and are a giant PITA to maintain or refactor. Throw in build systems and helper scripts and you quickly receive a giant pile of spaghetti. Personally I just switch to one the mentioned programming languages once it goes above a simple sequence of operations.

Personally I don't see how to improve it much without becoming a full blown programming language, at which point it would probably make more sense to just release a library for common automation tasks that is also composable. Maybe I'm just not the right target audience.

Re:What's New in POSIX 2024 by skydhash on Tuesday October 29, 2024 @02:21PM (#41987732)
The issue with your otherwise good reply is that someone are bringing expectations to an expert tool (programming languages, software, OS) and blidly assuming that everything will work as he thinks it should. Familiarity helps with learning, but shouldn’t replace it. Someone new to bash should probably start with a book.

And for bigger automation projects, there are lots of projects and programming languages that can help.

Re:What's New in POSIX 2024 by ManBeardPc on Tuesday October 29, 2024 @05:17PM (#41989495)
I agree it is an issue but it is how many people work and think. Most of the time they are not even wrong. "Hey, I have variables and loops, I know that!".

I would even make the case for expert tools being as unsurprising and familiar as possible unless there is a very good reason for them not to. Also they should be robust against misuse and guide the user towards good practices. There are always beginners, people that rarely need to use it, people that do programming as "just a job" and people that make mistakes because they are distracted, tired or just human. Something like "rm -r /" is a good reminder of that for many people.

Plus there are already a lot of tools required. Reading a book about every tool I have to use would be unpractical for most projects. Maybe more expert tools should just be tools. The same way I can now just use Ubuntu and get a working desktop system including drivers for most common hardware. If I compare that to the past where I installed a Linux distribution and then found out I lack a driver for my network card but I need to download it from the internet... I still can modify my system if I need to, but it's nice that I don't have to. I think we can do similar things with many parts of development and free some capacity for other tasks.

Re:What's New in POSIX 2024 by blueflow on Tuesday October 29, 2024 @04:15AM (#41980798)
Someone needs to come up with a interactive shell first, one that is comparable in usability. Then we can think about replacing the unix shell.

I tried both python and lua interactively, but they are a pain when it comes to handling files. You have to type much more to get the same things done.

Re:What's New in POSIX 2024 by anal_reactor on Tuesday October 29, 2024 @04:38AM (#41980952)
The bigger issue is the sheer momentum of Unix shell. Even if you come up with an alternative that is better by every objectively measurable metric, it's still going to be a monumental task to have it packages with commonly used distros. Kinda like the "why can't the US switch to the metric system" problem.
Re:What's New in POSIX 2024 by azalemeth on Tuesday October 29, 2024 @05:53AM (#41981740)
There's a direct cost in money, time and lives that has come from the US's adherence to their US Customary Units (which are often different to the old imperial units). People have literally died because of the confusion caused by having multiple systems of units in common use with ambiguous names (degrees, gallons, etc). Each year industry worldwide spends an enormous amount of money indirectly precisely because of this problem and it's still incredibly unlikely to be fixed within my lifetime.

Bash-alternatives that are not completely compatible frankly just don't have a chance.

Re:What's New in POSIX 2024 by blueflow on Tuesday October 29, 2024 @04:44AM (#41980987)
People already use different shells, mksh, fish, and so on. With fish there is a non-posix shell in wide use.
Re:What's New in POSIX 2024 by oguz-ismail on Tuesday October 29, 2024 @04:47AM (#41980999)
>wide use

Five people around the globe isn't wide use.

Re:What's New in POSIX 2024 by blueflow on Tuesday October 29, 2024 @04:51AM (#41981016)
I'm sure you might get more than 5 people on HN replying to you that they are using fish right now. Say something discrediting about fish and they show up.
Re:What's New in POSIX 2024 by fragmede on Tuesday October 29, 2024 @06:07AM (#41981905)
Heh, reminds me of how to get help with Linux back in the day. If you directly asked for help, you'd be told to RTFM. If you stayed confidently that Windows could do something and that Linux sucks because it can't, you'd get users tripping over themselves with details and instructions,'just to prove you wrong.

Human psychology is fascinating!

Re:What's New in POSIX 2024 by stephenr on Tuesday October 29, 2024 @07:25AM (#41982740)
If it isn't distributed out of the box with every nix-like OS, it inherently isn't* “better by every objectively measurable metric" - distribution of a common, stable standard is a huge benefit in and of itself.
Re:What's New in POSIX 2024 by blueflow on Tuesday October 29, 2024 @08:09AM (#41983054)
> distributed out of the box with every nix-like OS,

Python and lua are pretty close to that.

Re:What's New in POSIX 2024 by stephenr on Tuesday October 29, 2024 @08:51AM (#41983335)
> Python and lua are pretty close to that.

Python maybe often installed by default but it's definitely not an essential/required package "out of the box" on every install. Also, in a thread where one topic is how POSIX shell handles whitespace in filenames, it's hilarious (not in a good way) that someone suggests a language that handles whitespace the wrong way in it's own code. Yes, significant whitespace is objectively wrong.

What OS/distro is Lua included on out of the box? That doesn't mean "available in a package". I mean literally included in every single install and cannot reasonably be omitted?

Regardless of the availability, the parent comment says

> better by every objectively measurable metric

Neither Python nor Lua are "better" than shell, at the types of things shell is commonly used for - they're objectively worse.

Re:What's New in POSIX 2024 by blueflow on Tuesday October 29, 2024 @09:13AM (#41983523)
Lua gets onto every other Linux distro as dependency of some base system component. For example, rpm or pipewire depend on lua. Ubuntu and Debian ship with pipewire per default.

You should use the word "objectively" less.

Re:What's New in POSIX 2024 by stephenr on Wednesday October 30, 2024 @04:50AM (#41993017)
> Lua gets onto every other Linux distro

Just FYI, there are UNIX-like, POSIX compatible systems that are not a Linux distro.

> rpm or pipewire depend on lua. Ubuntu and Debian ship with pipewire per default.

Pipewire? Do you mean this? https://packages.debian.org/bookworm/pipewire

That isn't even close to "installed on every system". Best I can tell from the reverse dependencies, it's required for some Gnome Remote Desktop tool, and best I can tell, it doesn't rely on Lua anyway (at least on Debian).

> You should use the word "objectively" less.

I specifically used the word objectively, because the original comment that I replied to, said this:

> better by every objectively measurable metric

Re:What's New in POSIX 2024 by blueflow on Wednesday October 30, 2024 @05:25AM (#41993184)
pipewire -> wireplumber -> libwireplumber -> liblua

Pipewire being the Pulseaudio replacement from Redhat.

Bookworm is probably the last Debian without :P

Re:What's New in POSIX 2024 by stephenr on Wednesday October 30, 2024 @06:27AM (#41993465)
> Pipewire being the Pulseaudio replacement from Redhat.

Right, so it's a desktop package that ultimately will be installed on about 1% of all Linux machines because the vast majority are servers without a desktop environment.

Also worth pointing out: liblua on Debian at least, is the shared library. It's not the binary to execute standalone Lua scripts.

Re:What's New in POSIX 2024 by blueflow on Wednesday October 30, 2024 @08:02AM (#41993950)
This this like a game where you come up with bullshit and i have to come up with the facts to rectify it? RHEL/centOS have more than 1% market share alone.

Check your own installs and tell me if you find some that dont have liblua or libluajit.

For the library thing: I said "Python and lua are pretty close to that." earlier. I did not say that they have interpreters ready everywhere. But if the language core is already installed on a large fraction of machines, then adding the interpreter is not a big cost.

Re:What's New in POSIX 2024 by stephenr on Wednesday October 30, 2024 @09:16AM (#41994451)
> already installed on a large fraction of machines

So far you've presented no evidence of this though, just that it's used by a new desktop-focused package.

All linux desktops over the last 30 years is not even a "large fraction" of total Linux installs, much less the ones that have already migrated to this new audio system.

> adding the interpreter is not a big cost

It's nothing to do with cost. It's about "how do I know this will absolutely 100% run on any POSIX machine I throw it on without any extra steps".

Remember the argument here is about something that is claimed to be "objectively better" than Shell. The ubiquitous nature of POSIX shell is a huge barrier for any possible competitor, and saying "well you just need to install it" just defeats the purpose. You might as well write it in fucking java and say "well you just need to install a JVM".

Edit to Add: a good number of systems I manage do have liblua installed... because HAProxy requires it, and those systems have HAProxy installed. Not because it was installed as part of the base OS or even a default group of packages.

Incidentally, HAProxy and thus liblua were installed on those systems by infrastructure management that's implemented as shell script. So what kind of chicken and egg argument do we need to have here about how exactly I can run a Lua script to install Lua?

Re:What's New in POSIX 2024 by blueflow on Wednesday October 30, 2024 @01:09PM (#41997632)
> a good number of systems I manage do have liblua installed

/thread

Re:What's New in POSIX 2024 by consteval on Tuesday October 29, 2024 @09:41AM (#41983752)
Even outside of distribution, python and lua aren't objectively better. For starters, they're much more verbose.
Re:What's New in POSIX 2024 by blueflow on Tuesday October 29, 2024 @09:58AM (#41983895)
I just said that, scroll up.
Re:What's New in POSIX 2024 by Netch on Saturday November 02, 2024 @07:40AM (#42025749)
OK let them add an explicit check to standard tools, and/or to open(), mkdir(), etc. with O_PORTABLECHARS. And an environment option to disable this check.

Why they force the restriction at syscall level?

Re:What's New in POSIX 2024 by nly on Tuesday October 29, 2024 @05:35AM (#41981452)
Oil shell?

https://www.oilshell.org/

Compatible with most bash scripts

Re:What's New in POSIX 2024 by throw16180339 on Tuesday October 29, 2024 @09:57AM (#41983884)
I certainly have my complaints about Powershell, but it's got pretty good coverage, decent documentation, and cross platform support.
Re:What's New in POSIX 2024 by felixgallo on Tuesday October 29, 2024 @10:14AM (#41984040)
if it weren't so irregular, inconsistent, spotty and tasteless, it'd be a great option.
Re:What's New in POSIX 2024 by throwaway19972 on Tuesday October 29, 2024 @04:17AM (#41980818)
> the design of Unix shell is bonkers

Compared to what?

Re:What's New in POSIX 2024 by mdavid626 on Tuesday October 29, 2024 @05:45AM (#41981616)
Powershell?
Re:What's New in POSIX 2024 by poincaredisk on Tuesday October 29, 2024 @07:04AM (#41982553)
PowerShell designer could learn from decades of programming language progress and especially shell usage. They could improve many aspects indeed. This doesn't mean that the original design is "bonkers", only that it's not perfect.
Re:What's New in POSIX 2024 by pjmlp on Wednesday October 30, 2024 @09:45AM (#41994661)
The way Powershell works is largely based on what the computing world was doing with shells outside Bell Labs, at IBM, Xerox, and others places, exactly at similar timeframe as UNIX was happening.
Re:What's New in POSIX 2024 by mdavid626 on Thursday October 31, 2024 @03:17AM (#42004215)
Can you give examples of what should be improved in PowerShell?
Re:What's New in POSIX 2024 by oguz-ismail on Tuesday October 29, 2024 @05:51AM (#41981725)
Verbosity is a huge problem there
Re:What's New in POSIX 2024 by consteval on Tuesday October 29, 2024 @09:44AM (#41983772)
Modern programming language designers have a bad relationship with verbosity. I don't know why they do this.

It's a lang for an interactive shell, typing literally translates to developer speed. I understand the want for clarity and maybe that's nice in large scripts, but the main goal is to be a shell. So, optimize for that. Also, you probably shouldn't be using powershell for large scripts anyway.

The only recent lang I've seen that has a handle on this is Rust. You can tell they put a lot of thought into having keywords be as short as possible while still being descriptive.

Re:What's New in POSIX 2024 by ggm on Tuesday October 29, 2024 @06:42AM (#41982308)
FoundTheCamelCaseConvert.

My God next you will say getopt() --longform is the bestest

Re:What's New in POSIX 2024 by throw16180339 on Tuesday October 29, 2024 @09:54AM (#41983857)
It's been years since I used Powershell, but IIRC there are shortcuts for the common commands, e.g. cat, ls, mv, rm, and such DTRT.
Re:What's New in POSIX 2024 by Diti on Tuesday October 29, 2024 @03:56PM (#41988628)
Those aliases are, I believe, only defined on Windows PowerShell (the closed-source version 5; not PowerShell 7). I wish those default aliases you mentioned weren’t a thing. Especially `curl` (people should use `iwr` instead), which is an alias of `Invoke-WebRequest`, because it makes the `curl.exe` shipped with Windows nearly undiscoverable.
Re:What's New in POSIX 2024 by dailykoder on Tuesday October 29, 2024 @04:29AM (#41980893)
Works on my machine!
Re:What's New in POSIX 2024 by akira2501 on Tuesday October 29, 2024 @04:31AM (#41980909)
> I haven't seen any other tool ever have so many pitfalls.

I haven't seen any other tool with so much general utility and availability.

> to loop over a string array and print its contents

Is incredibly easy in bash and bash like shells. As highlighted the issue is that tools like 'ls' don't create "a string array." They create one giant string that has to be parsed. The rules in the shell are different than in other languages but it /will/ do most of the parsing for you, or all of it, if you do it carefully.

This is a fine tradeoff. As evidenced by it's wide usage and lack of convincing replacements.

Re:What's New in POSIX 2024 by anal_reactor on Tuesday October 29, 2024 @04:43AM (#41980985)
> I haven't seen any other tool with so much general utility and availability.

> availability

That's the real reason why we use Unix shell. It's not good, but it's available. Like a cheap hooker.

> but it /will/ do most of the parsing for you, or all of it, if you do it carefully.

"It mostly works if you're careful" doesn't sound very convincing to me.

Re:What's New in POSIX 2024 by stephenr on Tuesday October 29, 2024 @07:21AM (#41982714)
> but it's available. Like a cheap hooker.

Username checks out.

Re:What's New in POSIX 2024 by akira2501 on Tuesday October 29, 2024 @01:57PM (#41987470)
> "It mostly works if you're careful" doesn't sound very convincing to me.

Would you rather write your own parser?

Re:What's New in POSIX 2024 by vbezhenar on Tuesday October 29, 2024 @04:36AM (#41980938)
The main problem is using text as a common format between different applications.

First: text is not well defined. Is it ASCII? Is it UTF-8? Some programs can spew UTF-32 with proper locale configured, it's a mess.

Second: encoding and decoding of objects to text is not defined at all. Those problems with filenames is just one example. Using newline as a separator is a natural thing that is easy to implement, yet it is wrong.

In my opinion two things should be done:

1. Standardise on UTF-8. No other encodings allowed.

2. Standardise on JSON. It is good enough to serve as universal exchange format, tools like `jq` exist for some time now.

So any utility must read and write JSON objects with some standard env set. And shells can be developed with better syntax to deal with JSON. This way you can write something like

`ps aux | while read row; do echo ${row.user} ${row.pid}; done`

Re:What's New in POSIX 2024 by aloisklink on Tuesday October 29, 2024 @06:59AM (#41982499)
POSIX does actually define what a "text file" is, but the definition is a bit unusual:

See https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1...

> 3.387 Text File

> A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character.

So, if you have some non-printable characters like BEL/␇/ASCII 0x07, that's still a text file.

(and I believe what bytes count as a valid character depend on your `LC_CTYPE`).

But the moment you have a line longer than {LINE_MAX} bytes (which can depend on which POSIX environment you have), suddenly your text file is now a binary file.

Re:What's New in POSIX 2024 by WJW on Tuesday October 29, 2024 @07:48AM (#41982900)
Kind of a weird definition indeed. One edge case: the definition states the file must contain characters, so presumably zero length files are out. But then how could you have zero lines?
Re:What's New in POSIX 2024 by rascul on Tuesday October 29, 2024 @08:31AM (#41983203)
An empty file is not hard to make. It's just a matter of creating the file and not writing to it.
Re:What's New in POSIX 2024 by WJW on Tuesday October 29, 2024 @08:56AM (#41983380)
Yes obviously. But the POSIX specification for a "text file" as above is that it contains characters, which an empty file by definition does not. So an empty file cannot be a text file if you read that specification strictly, and therefore you cannot have zero lines in a text file. As soon as you have a single character there is at least one line, and the amount of lines can only stay the same or grow from there.

The definition should read "one or more lines" instead or (probably better) specify that a text file contains "zero or more characters".

Re:What's New in POSIX 2024 by rascul on Tuesday October 29, 2024 @09:43AM (#41983768)
Ahh I see what you're saying. I misunderstood at first.
Re:What's New in POSIX 2024 by Ukv on Tuesday October 29, 2024 @10:19AM (#41984093)
POSIX defines a line as:

> 3.185 Line

> A sequence of zero or more non-<newline> characters plus a terminating <newline> character.

So a file with some characters but no trailing newline is reported by `wc -l` as having zero lines.

Re:What's New in POSIX 2024 by poincaredisk on Tuesday October 29, 2024 @07:02AM (#41982526)
>It is good enough to serve as universal exchange format, tools like `jq` exist for some time now.

Please don't use that underdefined joke of a spec. Define "PosixJson" and use that instead. Right now it's not even clear what the result of parsing {"a": 1234678901234567890} is. Is this a parse error? A bigint? A float/double? Quiet wraparound? Something else? I've seen all these behaviors in real world JSON implementations across different languages.

Re:What's New in POSIX 2024 by oneeyedpigeon on Tuesday October 29, 2024 @06:21AM (#41982091)
I think a lot of tools should support json as well as plain text. Probably the latter by default, and the former with a "-o json" or similar option. I'm fine with wc giving me `5`, I'd prefer that to `{ "characters": 5 }`.
Re:What's New in POSIX 2024 by matrss on Tuesday October 29, 2024 @07:11AM (#41982645)
JSON itself is bad for a streaming interface, as is common with CLI applications. You can't easily consume a JSON array without first reading it in its entirety. JSONL would be a better fit.

But then, how well would it work for ad-hoc usage, which is probably one of the biggest uses of shells?

Re:What's New in POSIX 2024 by arghwhat on Tuesday October 29, 2024 @06:10AM (#41981948)
What cursed madness have you hit that spits out UTF-32 under normal conditions?! That can only be a bug - UTF-32/UCS-4 never saw external use, and has only ever been used for in-memory fixed-width character representation, e.g. runes in Go.

You never have to worry about whether you're dealing with ASCII vs. UTF-8, but rather if you're dealing with UTF-8 vs. ISO-8859-1, or worse, Shift JIS or similar.

Re:What's New in POSIX 2024 by vbezhenar on Tuesday October 29, 2024 @06:55AM (#41982457)
I think that I hit that with Java:

    % java -Dfile.encoding=UTF-32 Test | hexdump -C
    00000000  00 00 00 48 00 00 00 65  00 00 00 6c 00 00 00 6c  |...H...e...l...l|
    00000010  00 00 00 6f 00 00 00 2c  00 00 00 20 00 00 00 77  |...o...,... ...w|
    00000020  00 00 00 6f 00 00 00 72  00 00 00 6c 00 00 00 64  |...o...r...l...d|
    00000030  00 00 00 0a                                       |....|
    00000034

From quick googling it seems that glibc does not support it, so it should not happen.
Re:What's New in POSIX 2024 by Netch on Saturday November 02, 2024 @07:37AM (#42025736)
> it seems that glibc does not support it

`iconv` does, and this is enough in common. Among with tons of eerie EBCDIC/whatever...

Re:What's New in POSIX 2024 by Netch on Saturday November 02, 2024 @07:35AM (#42025731)
> That can only be a bug - UTF-32/UCS-4 never saw external use

I regularly use `iconv -t utf-32be | hd` to look what a bizarre sequence is denoting yet another weird symbol like an itchy hedgehog.

And what is a real reason to disallow this?

Re:What's New in POSIX 2024 by ezoe on Tuesday October 29, 2024 @05:33AM (#41981420)
Don't even assume UTF-something is the only character encoding. There are so many existing character encodings before Unicode. It's still widely used.
Re:What's New in POSIX 2024 by anal_reactor on Tuesday October 29, 2024 @04:42AM (#41980974)
True, but this would be immensely difficult to pull off, because how do you convince other people to write programs that produce actual working JSON?
Re:What's New in POSIX 2024 by nly on Tuesday October 29, 2024 @05:33AM (#41981423)
The primary purpose of command line program output is to convey information to a human, not to other programs.

Command line scripting is supposed to be adhoc and hack.

Re:What's New in POSIX 2024 by mdavid626 on Tuesday October 29, 2024 @05:39AM (#41981501)
I disagree that it supposed to be adhoc and hack. Look at PowerShell.
Re:What's New in POSIX 2024 by anthk on Tuesday October 29, 2024 @08:54AM (#41983367)
That under limited OSes such as DOS. Under Unix, piping has been the philosophy.
Re:What's New in POSIX 2024 by consteval on Tuesday October 29, 2024 @09:38AM (#41983729)
There are exchange formats that are well-defined enough to be useful to many computers while also being readable enough to be traversed by human eyes. There's no reason to everything ad-hoc, you don't get much by that. You also control the shell itself - there's no reason you can't display object representations in a pretty way.
Re:What's New in POSIX 2024 by pif on Tuesday October 29, 2024 @05:57AM (#41981790)
> The main problem is using text as a common format between different applications.

If you can't get the immensity of the cleverness of Unix foundations, you should not talk about them.

That idea is what made it possible for you to type that sentence in the first place.

Re:What's New in POSIX 2024 by zelphirkalt on Tuesday October 29, 2024 @07:31AM (#41982789)
This should not be as downvoted as it is. In a way shell is broken. The brokenness is in that it requires each command to serialize and deserialize again, considering all the weird things that can happen with the "all is a string" kind of approach, instead of having a proper data interchange format or even sending objects to next steps in the pipeline. This behavior is what necessitates even thinking about the changes listed in the post. We wouldn't even have that problem, if the design of shell was better thought out. Now we are dealing with decades of legacy built on these shaky foundations. I hate to admit it, but seems at least this aspect Powershell got right, whatever one may think about the rest of it.
Re:What's New in POSIX 2024 by chasil on Tuesday October 29, 2024 @02:07PM (#41987574)
On my rhel7 system, the Debian dash shell is this large:

   $ ll /bin/dash
  -rwxr-xr-x. 1 root root 113536 Nov  5  2018 /bin/dash
I happen to have an old powershell installed:

  $ rpm -qi powershell | grep Size
  Size        : 126588370
A strict POSIX shell is always going to be vastly smaller, for many reasons.

I would prefer that the POSIX shell was an LR-parsed language, but you can't have everything.

Re:What's New in POSIX 2024 by nneonneo on Tuesday October 29, 2024 @05:18AM (#41981274)
Rather unfortunately, I happen to have a handful of files on my machine with newlines in them (the filenames were programmatically generated from a summary of their contents). I loathe the possibility that my shell tools are going to suddenly crash when confronted with these weird files, rather than just producing some slightly silly output. I wish we'd standardized the behaviour of just escaping such characters as `\n/\r` or `^J/^M`...
Re:What's New in POSIX 2024 by jraph on Tuesday October 29, 2024 @07:42AM (#41982858)
They did the right thing for this: make the tools fail on file creation, but not on existing files.

I guess it's still advisable to rename those files, I don't know how things like cp, mv or rsync will behave when copying such files in the future.

Re:What's New in POSIX 2024 by ykonstant on Tuesday October 29, 2024 @09:05AM (#41983454)
If your file system allows them, be careful with symlinks though!
Re:What's New in POSIX 2024 by jraph on Tuesday October 29, 2024 @12:06PM (#41985672)
Why, specifically?

I'm convinced we will need to be careful with symbolic links related to new line characters in filenames, but I'm curious of which specific aspect you had in mind.

Re:What's New in POSIX 2024 by ykonstant on Tuesday October 29, 2024 @03:47PM (#41988557)
Oh, nothing specific to newlines. Just, when you rename files to fix newlines, you need to check if they break symlinks pointing to them.

For instance, I had project folders for my individual research projects. In order to have a central repository of resources and not have copies of multi-megabyte pdfs in each folder, I put all referenced papers in a single directory and symlinked them for each project that needed them. Later, I wanted to rename the papers to remove newlines. The symlinks complicated this process quite a bit!

Re:What's New in POSIX 2024 by jraph on Wednesday October 30, 2024 @05:33AM (#41993228)
Ah, right, indeed :-)
Re:What's New in POSIX 2024 by nneonneo on Tuesday October 29, 2024 @10:02AM (#41983939)
No, they did not do the right thing:

> the following utilities are now either encouraged to error out if they are to create a filename that contains a newline, and/or encouraged to error out if they are *about to print a pathname that contains a newline* in a context where newlines may be used as a separator

It then proceeds to list a bunch of utilities including diff, file, find, grep, head, du, etc., none of which create files directly.

These utilities could be updated to reject newlines in file paths if they're going to print in a "newline delimited" form - but for some of these utilities, that's the only available form.

Re:What's New in POSIX 2024 by jraph on Tuesday October 29, 2024 @11:53AM (#41985451)
> error out if they are about to print a pathname that contains a newline in a context where newlines may be used as a separator

But that's already broken. This is a situation where filenames with newlines in them are indistinguishable from two filenames in outputs. So instead of producing subtly broken output, tools are encouraged (not forced) to explicitly fail with a lot of noise.

The "in a context where newlines may be used as a separator" part of this sentence is very important.

IIUC the tools are still allowed to succeed in non broken situations, for instance when a null separator is used and not a new line character. And I can't imagine the tools you listed will start breaking in situations that worked (apart from file creation - indeed this will likely start breaking, and new line characters in filename needs to be considered deprecated and things using them to be fixed).

This is strictly better IMHO (if one thinks that newlines in files are not worth the troubles given how things work in POSIX, especially the part where things are line-based and new line characters have quite some significance)

Re:What's New in POSIX 2024 by nasretdinov on Tuesday October 29, 2024 @05:58AM (#41981808)
The thing is, it's hard to predict what would happen to those scripts regardless... E.g. try naming your files "-rf" and see how many things break :)
Re:What's New in POSIX 2024 by tetha on Tuesday October 29, 2024 @07:55AM (#41982947)
I do enjoy "ls *; touch -- -lisah; ls *" as a fun little brainteaser for those uninitiated to this behavior.
Re:What's New in POSIX 2024 by redserk on Tuesday October 29, 2024 @07:24AM (#41982734)
If one really wanted to embrace chaos, introduce this as a new team file naming standard for "risk finding" files ;)
Re:What's New in POSIX 2024 by ykonstant on Tuesday October 29, 2024 @09:22AM (#41983602)
A correct script will have no problems with "-rf" or any other file name. I have (and recommend script writers make their own) a directory hierarchy of "dangerous" file names to test scripts.

For example, it contains a directory where all file and subdirectory names are in unary, consisting only of repetitions of the newline character. A correct script should be able to enumerate, access and modify files in there without issue.

Re:What's New in POSIX 2024 by nneonneo on Tuesday October 29, 2024 @10:12AM (#41984014)
export TMPDIR=" / "

to surprise the next person or script to do "rm -rf $TMPDIR/foo"...

Re:What's New in POSIX 2024 by ykonstant on Tuesday October 29, 2024 @06:11AM (#41981965)
In academia, I get (and used to create) pdfs with names like:

"On the number of

associative foobars

of degree blah -

Johnson and Anderson.pdf"

all the time. It is very convenient for non-technical academics to have a descriptive file name, and to be able to see it entirely in the navigator they use newlines.

Re:What's New in POSIX 2024 by jodrellblank on Tuesday October 29, 2024 @08:04AM (#41983010)
I don't know who "the Austin Group" mentioned in the article are, but how come they "could not find a single use-case for newlines in pathnames besides breaking naive scripts" when legitimate use-cases are so easy to find?

(And if they're that incompetent, why does the article imply they are worth quoting and listening to?)

Re:What's New in POSIX 2024 by gpderetta on Tuesday October 29, 2024 @09:22AM (#41983601)
It is [1] the joint working group that for the last 25+ years has been responsible for both the POSIX standard and the Single Unix Specification. It emerged after the UNIX wars as a consolidation of the various splintered UNIX standardization efforts (POSIX itself, X/OPEN, OSF, etc).

[1] https://en.wikipedia.org/wiki/Austin_Group

Re:What's New in POSIX 2024 by nilamo on Tuesday October 29, 2024 @11:01AM (#41984528)
Is that legitimate? A path name is just a unique identifier for a file, IMO it doesn't make sense to put a whole novel in there. If anything, a giant summary like that should be in the meta tags?
Re:What's New in POSIX 2024 by jodrellblank on Tuesday October 29, 2024 @01:02PM (#41986621)
In what way is it not legitimate? It's not an accident, bug or data corruption. Someone put it there for a reason, and it benefits their use case. That's as legitimate as it gets.
Re:What's New in POSIX 2024 by ykonstant on Tuesday October 29, 2024 @03:19PM (#41988321)
That's a core part of the problem: a path name is NOT just a unique identifier for a file. Desktop operating systems and their classical utilities conflate the "unique identifier" and whatever "displayed title" of a file though which the end user interacts with the file.

Users care about "titles" or "summaries" of files, not "filesystem identifiers"; as long as the two are conflated, non-technical users will use the identifier to write titles and thus make the file easy to locate in an interactive GUI. Meta tags are not even in the cognitive horizon of most people.

Re:What's New in POSIX 2024 by shthed on Thursday October 31, 2024 @05:55PM (#42011968)
Name one legit use case.
Re:What's New in POSIX 2024 by jodrellblank on Thursday October 31, 2024 @07:00PM (#42012409)
... the use case in the parent comment I was replying to.

And no I'm not going to copy that here for you to quip "that's not a legitimate use case". Make an effort to make a point and support it with better justification than "because I said so".

Re:What's New in POSIX 2024 by oneeyedpigeon on Tuesday October 29, 2024 @06:18AM (#41982044)
Oh god. I already get upset enough by spaces in a file name, although I realise that fight is basically lost now!
Re:What's New in POSIX 2024 by curt15 on Tuesday October 29, 2024 @07:23AM (#41982730)
Didn't Windows name "Program Files" with a space to force application developers to handle spaces in paths properly?
Re:What's New in POSIX 2024 by ape4 on Tuesday October 29, 2024 @08:18AM (#41983109)
Not to mention C:\Program Files (x86)
Re:What's New in POSIX 2024 by account42 on Tuesday October 29, 2024 @08:32AM (#41983215)
And C:\Programme and other localized variants to force people to go through the proper APIs instead of hardcoding paths.
Re:What's New in POSIX 2024 by pjmlp on Tuesday October 29, 2024 @08:37AM (#41983249)
In theory yes, in practice to this day many people don't bother how to learn how to deal with pathnames in a proper way.
Re:What's New in POSIX 2024 by inetknght on Tuesday October 29, 2024 @09:38AM (#41983726)
Top difficulties in computer science:

1. naming things

2. cache coherency

3. off-by-one errors

???

4. quoting pathnames

Re:What's New in POSIX 2024 by hawski on Tuesday October 29, 2024 @12:18PM (#41985880)
I would replace 4 with parameter expansion rules.
Re:What's New in POSIX 2024 by nneonneo on Tuesday October 29, 2024 @10:05AM (#41983965)
Eh, maybe. In practice I usually do all my moderately-heavy filesystem scripting in Python these days, for which pathname quoting is just a complete non-issue. Of course, I still use a shell for quick-and-dirty stuff, but usually only for pretty simple tasks where the simplest quoting setup ("$i") suffices.
Re:What's New in POSIX 2024 by mmcdermott on Tuesday October 29, 2024 @12:04PM (#41985632)
For the longest time you could get away with this in cmd:

> dir c:\progra~1

So if forcing people to handle spaces was the goal, it took a long time to force it.

Re:What's New in POSIX 2024 by arp242 on Tuesday October 29, 2024 @12:50PM (#41986435)
I'm pretty sure that still works. I forgot the exact scenario, but my Windows CI on GitHub Actions output shorte~1 pathna~1 like that in a script just a few months ago. On one hand, the backwa~1 compati~1 is nice. On the other hand, there's just so much depreca~1 cruft that keeps popping up even on contemp~1 systems.
Re:What's New in POSIX 2024 by InfiniteRand on Tuesday October 29, 2024 @04:50PM (#41989244)
I just got used to installing things I need to interact with in a program into a folder named C:\workspace
Re:What's New in POSIX 2024 by enriquto on Tuesday October 29, 2024 @06:40AM (#41982277)
As a fellow spaces-in-filenames-hater, the fight is not lost. We are on the brink of winning it; it's just a mount option away!

While we cannot avoid that people hit the spacebar when writing a filename on a gui, this does not mean at all that the resulting filename itself need contain a plain space character. Those spaces can and should be transparently translated to non-breaking space characters at some point. Maybe by the gui itself, or more robustly by the filesystem. This would make everybody happy: gui users and naive shell script writers.

Re:What's New in POSIX 2024 by poincaredisk on Tuesday October 29, 2024 @06:54AM (#41982455)
>Those spaces can and should be transparently translated to non-breaking space characters at some point

Why? This just introduces more complexity and interoperability headaches for seemingly no reason.

Re:What's New in POSIX 2024 by enriquto on Tuesday October 29, 2024 @07:13AM (#41982667)
> Why?

In order to preserve the sacrosanct simplicity of naive shell scripts. Seems like a very noble goal to me.

The only unexpexted compexity arises when you want to deal with filenames having mixed spaces and nbsps. But I'd say that people who do that had it coming.

Re:What's New in POSIX 2024 by alexvitkov on Tuesday October 29, 2024 @07:53AM (#41982934)
If you want simple shell scripts to work, make an actually good shell language without all the footguns.

The filesystem is way more important than /bin/sh and and any complexity added there will trickle down to all programs, not just shell scripts.

It's not worth adding hacks on the FS to patch defects in poorly written shell scripts (which are being replaced en masse with python/nodejs/even weirder yaml files/systemd units/etc... anyways)

Re:What's New in POSIX 2024 by ksp-atlas on Tuesday October 29, 2024 @09:08AM (#41983483)
nushell uses real lists for things which means you don't need to care about seperators except when dealing with external system things
Re:What's New in POSIX 2024 by consteval on Tuesday October 29, 2024 @09:33AM (#41983690)
Whitespace in filenames in general is difficult to deal with. Many, maybe most, programs get it wrong. It's not just about shell scripts, many GUI programs fail to handle those files properly too.
Re:What's New in POSIX 2024 by alexvitkov on Tuesday October 29, 2024 @10:06AM (#41983978)
When GUI programs mishandle filenames with spaces, IME it's usually because they spawn a subshell in a naive way (system("rm " + filename)).

To mishandle spaces you have to split an input w/ filenames by whitespace, which is not that common of an operation outside of a shell.

Re:What's New in POSIX 2024 by InfiniteRand on Tuesday October 29, 2024 @04:52PM (#41989274)
My favorite file+space issues is spaces at the end of file names, especially when you copy and paste text, or text gets trimmed from an input box, or the person forgets to trim space from an input box...
Re:What's New in POSIX 2024 by nradov on Tuesday October 29, 2024 @11:21AM (#41984776)
The vast majority of Windows and MacOS programs get it right.
Re:What's New in POSIX 2024 by consteval on Wednesday October 30, 2024 @10:07AM (#41994971)
No, no they don't - you just don't notice when they get it wrong, and you also don't name your files stupid things (I imagine).

If you actually test this, you'll realize a ton of Windows programs get it wrong.

Also, in general this is a poor argument. The goal of Linux isn't to be as much like Windows as possible, because Windows sucks ass. Nobody in their right mind would use Linux if it was just Windows but, presumably, shittier. The entire appeal of Linux is that it isn't Windows, and it isn't MacOS.

Re:What's New in POSIX 2024 by arp242 on Tuesday October 29, 2024 @12:54PM (#41986507)
Eh? It's really not a bother in pretty much any programming language, and you don't really need to do anything special for it. I don't know any program that has any problems with it.

Even zsh has fixed this. It's just /bin/sh and bash that are annoying.

Re:What's New in POSIX 2024 by lifthrasiir on Tuesday October 29, 2024 @08:54AM (#41983360)
Simplicity doesn't always mean stupidity. The simple but functional shell that correctly handles whitespaces without much hassle was already available since 90s, namely rc which is also found in Plan 9. Adopting rc's string concatenator `^` in POSIXy shells shouldn't be too hard.
Re:What's New in POSIX 2024 by vbezhenar on Tuesday October 29, 2024 @07:01AM (#41982518)
Yep, works today:

    sh-3.2$ f='Hello world'
    sh-3.2$ echo $f
    Hello world
    sh-3.2$ for i in $f; do echo $i; done
    Hello
    world

    sh-3.2$ f='Hello\xC2\xA0world'
    sh-3.2$ echo $f
    Hello world
    sh-3.2$ for i in $f; do echo $i; done
    Hello world
Re:What's New in POSIX 2024 by stouset on Tuesday October 29, 2024 @07:47AM (#41982893)
Just always quote variable interpolation and you will never have problems.

    sh-3.2$ f='Hello world'
    sh-3.2$ echo "${f}"
    Hello world
    sh-3.2$ for i in "${f}"; do echo "${i}"; done
    Hello world
    sh-3.2$
Re:What's New in POSIX 2024 by chasil on Tuesday October 29, 2024 @10:35AM (#41984248)
It would be really nice if there was a mount option that would quietly remove spaces in filenames, or convert them to an underscore.

If I had it, I would use it today.

Re:What's New in POSIX 2024 by jonhohle on Tuesday October 29, 2024 @10:25AM (#41984168)
Convince (force?) your team to use make and soon everyone will forget spaces in file names are even a thing!
Re:What's New in POSIX 2024 by taneliv on Tuesday October 29, 2024 @10:58AM (#41984497)
My team already uses `make` but there's no reason for me to run it in my Downloads folders. File names in there are sometimes wild. Yet I expect command line tools to work with them. If they will cease to do so, I will have to start using non-POSIX variants of those tools, I guess.
Re:What's New in POSIX 2024 by Flimm on Tuesday October 29, 2024 @06:56AM (#41982468)
How do these non-technical academics even create a PDF file with a name like that?
Re:What's New in POSIX 2024 by ykonstant on Tuesday October 29, 2024 @07:11AM (#41982649)
Right click, rename, enter, enter, enter (until the entire file name is visible on the box)? That's how I did it when I used Windows.
Re:What's New in POSIX 2024 by zelphirkalt on Tuesday October 29, 2024 @07:24AM (#41982736)
Doesn't <enter> just confirm the typed input for the filename and finish the renaming? How does that insert newlines?
Re:What's New in POSIX 2024 by ykonstant on Tuesday October 29, 2024 @07:33AM (#41982794)
Shrug, I last used windows with Windows 7, so you are probably right. That being said, at least two of the students I am currently tutoring are on XP and one of my colleagues as well :D
Re:What's New in POSIX 2024 by pino82 on Tuesday October 29, 2024 @08:50AM (#41983328)
No, it was always this way.
Re:What's New in POSIX 2024 by ykonstant on Tuesday October 29, 2024 @09:01AM (#41983425)
Right, I just remembered the main way to create those filenames: open the pdf, select and copy the title, close, rename the file and paste from clipboard.
Re:What's New in POSIX 2024 by astroid on Tuesday October 29, 2024 @04:23PM (#41988941)
Yes - I just tested on Win10+11 because I thought "there is no way I didn't accidentally do something like this on accident... and I would have remembered seeing a new line in my file name when I made that mistake."

I just opened a folder in file explorer, clicked 'rename' and then tried the following combinations: Enter L Ctrl + Enter L Alt + Enter Win + Enter R Ctrl + Enter R Alt + Enter

None of them let me put new lines in the filename - it either did nothing, or 'closed' the rename view.

Re:What's New in POSIX 2024 by abenga on Tuesday October 29, 2024 @08:10AM (#41983059)
I don't know if this is a Linux thing, but when renaming a file, when I press enter, I apply the new name, the file manager doesn't add a newline.
Re:What's New in POSIX 2024 by ykonstant on Tuesday October 29, 2024 @09:54AM (#41983860)
I am interested in hearing the rationale for downvotes explicitly. I am describing a reality that exists and must be taken into account. Why are you downvoting?
Re:What's New in POSIX 2024 by stouset on Tuesday October 29, 2024 @07:44AM (#41982872)
Dude, just fix the filenames.
Re:What's New in POSIX 2024 by funcDropShadow on Wednesday October 30, 2024 @05:13AM (#41993120)
Of course, there is an xkcd for that: https://xkcd.com/1172/
Re:What's New in POSIX 2024 by ezoe on Tuesday October 29, 2024 @05:31AM (#41981376)
Don't assume UTF-8 is the only character encoding used in the wild. There are character encoding with leading bytes not easily detectable like UTF-8.
Re:What's New in POSIX 2024 by arghwhat on Tuesday October 29, 2024 @06:04AM (#41981864)
In 2024, if you don't get the correct result decoding a text as UTF-8, the bug is the text, not the decoding. And luckily, adoption of UTF-8 in the past 30+ years have gone will enough that you don't need to worry.

Caveats for cursed hardware standards demanding two-byte encodings like USB.

Re:What's New in POSIX 2024 by poincaredisk on Tuesday October 29, 2024 @06:57AM (#41982484)
I hope you're happy in your ivory tower, but I personally work with a lot of files with other encoding, most often that weird utf16 (Windows), sometimes also legacy files with different ANSI encoding. Declaring "my decoder is fine, it's the text that is buggy" is not going to score a lot of points with my boss and clients.
Re:What's New in POSIX 2024 by zelphirkalt on Tuesday October 29, 2024 @07:27AM (#41982753)
Shouldn't hurt to tell clients to right their weird proprietary software originated encodings though.
Re:What's New in POSIX 2024 by arghwhat on Tuesday October 29, 2024 @12:15PM (#41985819)
The only valid reason for still having files stored in legacy ANSI encodings is that their only use is input to software that has not been maintained for ~30 years and cannot be updated. That's fine because they're just binary inputs in a closed ecosystem that no one touches.

But if they are supposed to be treated as text, then yes it's the text that's buggy - they should just be converted to UTF-8 once and have the originals thrown away.

UTF-16 is something that Microsoft has cursed us with by inserting it into specifications (like USB) so that we cannot get rid of it, even if it never made any sense what so ever. But those are in effect explicit protocols with a hard contract, very different from something where you would "assume an encoding".

Re:What's New in POSIX 2024 by 1oooqooq on Tuesday October 29, 2024 @07:22AM (#41982723)
why people assume utf8 had only know locale encoding still?

you're probably guilty of the sin you preach and is showing wrongly decoded utf8 and don't even know.

Re:What's New in POSIX 2024 by account42 on Tuesday October 29, 2024 @08:22AM (#41983136)
Their proposed solution is not compatible with reality though where POSIX does not get to define what kind of files exist on filesystems you need to work with.

All they did is introduce new error cases in C programs while not actually fixing anything for shell scripts.

If anything, it's going to result in more exploits as people write shell scripts with the assumption that newlines cannot appear in filenames.

Re:What's New in POSIX 2024 by quotemstr on Tuesday October 29, 2024 @10:54AM (#41984448)
In the real world, nobody writes shell scripts that handle newlines in filenames.
Re:What's New in POSIX 2024 by account42 on Tuesday October 29, 2024 @12:11PM (#41985752)
I do. Single files are handled with quotes around arguments just fine. For lists of files you need to use NUL as a separator. That's not really hard to do once you are aware of the problem but ergonomics could be better - which is something useful that POSIX could change.
Re:What's New in POSIX 2024 by quotemstr on Saturday November 02, 2024 @05:50PM (#42029456)
Most shell scripts that correctly handle newlines do so by accident.
Re:What's New in POSIX 2024 by account42 on Monday November 04, 2024 @03:39AM (#42039676)
Because they don't deal with lists of files so this whole issue doesn't apply? Yes. Even more reason to just fix the shell scripts where this matters instead of trying to impose additional restrictions on filesystems with many decades of history.
Re:What's New in POSIX 2024 by zokier on Tuesday October 29, 2024 @08:54AM (#41983370)
But they did not make old code correct. Filenames are still allowed to contain newlines. Shell scripts still need to be prepared to deal with that. Nothing really changed, they just added a feel-good half-measure.
Re:What's New in POSIX 2024 by quotemstr on Tuesday October 29, 2024 @10:53AM (#41984444)
It's a step in the right direction. You have to understand that for decades a vocal group of Unix die-hards has opposed any limitations whatsoever on the bytewise content of file names. The newline restriction in this latest version of POSIX may be modest, but it represents a dam breaking. When (obviously) the sky doesn't fall, the next version of POSIX will have a lot more filename cleanup.
Re:What's New in POSIX 2024 by rixed on Wednesday October 30, 2024 @01:39AM (#41992166)
Next step is to forbid newlines from file content itself, to fix conforment json parsers ?
Re:What's New in POSIX 2024 by janderland on Tuesday October 29, 2024 @12:53PM (#41986487)
This is pretty standard for a human run system. Gotta make the human feel good about an idea before they can do said idea.

If you’re not familiar with humans, there are several manuals available online.

Re:What's New in POSIX 2024 by hwc on Tuesday October 29, 2024 @01:45PM (#41987323)
Now do that with all whitespace!