News for hackers. Stuff that matters.
Re:Posix Has Become Outdated (2016) [pdf] by cperciva on Saturday February 11, 2017 @04:21AM (#13621882)
The original goal of the POSIX standard was application source code portability. However, modern applications are no longer being written to standardized POSIX interfaces.

Speak for yourself.

Re:Posix Has Become Outdated (2016) [pdf] by coldtea on Saturday February 11, 2017 @08:30AM (#13622504)
>Speak for yourself.

Actually the inverse: don't speak for yourself, because it's irrelevant.

What the authors did was try to speak for what happens at large -- which is what matters. Outliers will always exist, but their team analyzed thousands of packages across three different OS environments to come to conclusions.

It's not about there being counter-examples, it's about what they saying being the norm.

Re:Posix Has Become Outdated (2016) [pdf] by cperciva on Saturday February 11, 2017 @12:28PM (#13623583)
They established that many applications are not POSIX-compliant. They did not establish that modern applications are not POSIX-compliant.
Re:Posix Has Become Outdated (2016) [pdf] by mwfunk on Saturday February 11, 2017 @02:01PM (#13624122)
I don't know what you're referring to with that distinction. AFAICT they weren't testing for POSIX compliance but rather where the POSIX layer was being used in practice, which statistically was by system-level frameworks rather than the applications themselves on the OSes they tested.

The notable change between now and 25 years ago is that the POSIX layer tends to be used at the lower levels of system frameworks, while the programs that run on those OSes tend to be written on top of higher level frameworks and do not directly use the POSIX APIs.

This is fine and makes sense given how much has changed in the last 25 years, but given that the POSIX layer is no longer being used (statistically speaking) for what it was originally designed for, it raises questions about to what degree that design is still valid, and whether or not it should be revisited to better suit what it's actually being used for.

I don't think anyone has any answers, but it's always good to question assumptions and never let anything in tech become a sacred cow. Maybe POSIX is fine for its current role, but that doesn't mean people shouldn't think about it.

Disclaimer: I skimmed the article pretty aggressively, so I may have misunderstood something.

Re:Posix Has Become Outdated (2016) [pdf] by yarou on Saturday February 11, 2017 @02:09PM (#13624163)
I'm a little confused by the distinction. For example, if I use a framework that abstracts away pthreads from me, but still uses pthreads in its implementation of threading, is the application I write still POSIX-compliant?
Re:Posix Has Become Outdated (2016) [pdf] by mwfunk on Saturday February 11, 2017 @02:34PM (#13624295)
No, the application would not be POSIX-compliant, but it would be running on a POSIX-compliant OS. I was confused by the comment I replied to because it sounded like he was saying that POSIX compliance was more likely in "modern" applications, but if anything the opposite is the case.

POSIX compliance is much more common in consumer OSes nowadays, to the point that every major desktop and mobile platform is built on a POSIX-compliant kernel and runtime except for Microsoft's offerings. In that sense POSIX has taken over the non-Microsoft world, but now it's being used as a much lower-level compatibility layer for system-level frameworks rather than what it was originally designed for.

Even Windows has had multiple POSIX implementations over the years, they were just built as compatibility layers on top of the native APIs. So POSIX-compliant operating systems are almost universal at this point, but POSIX-compliant applications/daemons/etc. (all the programs that run on those POSIX OSes) are rarer than ever.

Re:Posix Has Become Outdated (2016) [pdf] by cperciva on Saturday February 11, 2017 @03:33PM (#13624651)
it sounded like he was saying that POSIX compliance was more likely in "modern" applications

Yes. Part of this is because more operating systems are approaching POSIX-compliance; but the number of patches we need to port random Linux code to run on FreeBSD has dropped tremendously over the past 20 years.

Re:Posix Has Become Outdated (2016) [pdf] by coldtea on Saturday February 11, 2017 @06:32PM (#13625518)
>is the application I write still POSIX-compliant?

Obviously not, since you can't go and compile it on another POSIX system -- you are "abstraction compliant" (or rather "abstraction dependent" now.

Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @04:35AM (#13621917)
Try to write a modern iOS, OS X, Windows, Android, ChromeOS UWP with standardized POSIX interfaces.
Re:Posix Has Become Outdated (2016) [pdf] by geezerjay on Saturday February 11, 2017 @10:09AM (#13622912)
> Try to write a modern iOS, OS X, Windows, Android, ChromeOS UWP with standardized POSIX interfaces.

You're confusing popularity with being outdated.

It's irrelevant if some vendors intentionallly go out of their way to avoid implementing any standard interface which, in the very least, can't be controlled by them.

And it's also absurd that you're quoting platforms which even restrict which programming language can be used to write applications purely due to their business strategies.

Re:Posix Has Become Outdated (2016) [pdf] by nickpsecurity on Saturday February 11, 2017 @11:08AM (#13623168)
"You're confusing popularity with being outdated."

POSIX was supposed to be popular to the point of default way that apps were written. It failed. Whereas there's quite a bit of portable apps on (insert framework here). They succeeded with many running on non-UNIX OS's, too. You'll need those or similar things in your code instead of POSIX for truly, portable apps. That's his point.

Re:Posix Has Become Outdated (2016) [pdf] by felixgallo on Saturday February 11, 2017 @12:11PM (#13623477)
> POSIX was supposed to be popular to the point of default way that apps were written. It failed.

So POSIX was a long series of attempts to codify baseline Unix practice that arose because in the '87 time period there were tens of small, warring, incompatible Unix distributions ( cf. http://www.ugu.com/sui/ugu/show?ugu.flavors ), and systems software vendors could not write software that worked across all of these platforms properly.

The original post report is a cri de coeur call to action written in POSIX magazine attempting to oversell the story with a clickbait title in order to urge further POSIX work. This thread and others is loaded with people that have only read the title and have their own half-formed opinions.

There was never any idea that all apps everywhere would be written with POSIX. That's a ridiculous reading. I was there at the time. It was just an effort to get all of the Unix vendors on the same page so that the big commercial non-Unix OS vendor and their salespeople wouldn't eat the lunch of the high end workstation and server market.

And it succeeded phenomenally. With POSIX as a more-or-less stable substrate, it became possible to run, e.g., gnu's entire platform on almost anything that was POSIX-compliant or close; e.g. SunOS, Solaris, and eventually Linux, with minimal effort. It became possible for software like vi, emacs, mysql, sqlite to exist. Sysadmins could write shell scripts without anything like the kind of portability fear they had in the past.

Uncoincidentally, this continues today; all of the non-Windows software you're using is based on POSIX standards. Every phone is packed with POSIX-foundationed databases, daemons and sometimes even userlands. And of course there's Linux which, despite the continuing efforts of Red Hat, is about as POSIX as it comes.

Read the box at the bottom of page 9 of the original post. POSIX won, totally and unconditionally.

Re:Posix Has Become Outdated (2016) [pdf] by geezerjay on Saturday February 11, 2017 @12:19PM (#13623526)
> I was there at the time. It was just an effort to get all of the Unix vendors on the same page so that the big commercial non-Unix OS vendor and their salespeople wouldn't eat the lunch of the high end workstation and server market.

Precisely.

Additionallly, criticizing the technical merits of an international standard established in 1997 is a disingenuous ordeal. If anyone actually has any meaningful and tangible technical issue to point out regarding POSIX, they should simply put forth an implementation that actually addresses the issues and use it as a basis for a discussion to update the standard.

Standards are meant to be used as references, and therefore are meant to be updated when the need arises.

Re:Posix Has Become Outdated (2016) [pdf] by nickpsecurity on Saturday February 11, 2017 @06:21PM (#13625462)
As I said go Async, I think there's two conversations going on in these threads: POSIX for UNIX portability; general portability. POSIX succeeded on prior but not latter as it wasnt designed for it. Today, we have portability frameworks that can do more than POSIX on more OS's. They definitely win on general portability with them worth consideration for UNIX portability. That's my position.

Far as early days, you might be able to help me out on a point I have few sources on. One history of Xenix posted here claimed its influence and design choices had strong influence on adoption of features that latter led to POSIX. Source was biased so hard to assess. What's your take on Xenix's influence on adoption of standard or POSIX-like API's?

Re:Posix Has Become Outdated (2016) [pdf] by felixgallo on Saturday February 11, 2017 @09:33PM (#13626308)
I bet your source was right. I don't have inside knowledge, but as one of the first Unixes to try to make a serious business out of it, and as an arguable progenitor of the SysV approach, it wouldn't surprise me.

That said, POSIX was never designed for general portability. Just as a seal of unifying approval and sanity for existing practice; far more descriptive than prescriptive.

Re:Posix Has Become Outdated (2016) [pdf] by geezerjay on Saturday February 11, 2017 @11:25AM (#13623241)
> POSIX was supposed to be popular to the point

If you read up on why companies such as Microsoft make it their point to break and avoid standards, you would understand why your comments regarding "popularity" are moot.

Business strategies don't make or break technical merits.

Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @11:36AM (#13623309)
Are you also speaking about all those APIs in GNU/Linux that aren't part of any other UNIX clone?
Re:Posix Has Become Outdated (2016) [pdf] by geezerjay on Saturday February 11, 2017 @12:09PM (#13623471)
You're confusing things. You don't break a standard when you write a function or even a whole library.
Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @02:23PM (#13624236)
So Microsoft, Google and Apple do it, it is bad.

Linux does it, it is me that doesn't understand, I see.

Re:Posix Has Become Outdated (2016) [pdf] by AsyncAwait on Saturday February 11, 2017 @04:13PM (#13624826)
Linux is mostly POSIX compliant. It has layers on top that are not POSIX, but that doesn't mean it is not POSIX or that you can't expect your POSIX apps to work on Linux.
Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @05:07PM (#13625127)
Not at all, it means that many mistakenly use Linux syscalls as synonym for POSIX, thus making the same lock-in to GNU/Linux as Google, Apple and Microsoft do to their own systems.

*BSD devs jump of joy having to port GNU/Linux applications that depend on D-BUS, systemd, or any other Linux specific APIs.

Re:Posix Has Become Outdated (2016) [pdf] by AsyncAwait on Saturday February 11, 2017 @05:40PM (#13625285)
Given that *BSDs also have many of their own extensions, (ie openbsd has many security extensions to the POSIX standard), that isn't the best example of a defense.
Re:Posix Has Become Outdated (2016) [pdf] by nickpsecurity on Saturday February 11, 2017 @11:32AM (#13623286)
I would not if the goal of a standard is uptake by OS and application developers. Wide usage of a product = popularity. It didn't achieve the popularity it sought. This is independent of what businesses like Microsoft were doing.

Microsoft's strategy was to create a permanent dependence on them for long-term profitability. They had too many tactics for that to cover here. Once open standards proliferated, they subverted them too with proprietary extensions under "Embrace, Extend, Extinguish" strategy. Companies buying into their solutions and using non-standard features paid the price.

Regardless, people still created apps that worked on such proprietary platforms and UNIXen via portability frameworks or libraries. Some products using them became quite popular (eg Apache, Netscape/Firefox) with some of the frameworks themselves becoming popular (eg Qt, GTK, wxWindows, Tk). POSIX's got into enough stuff that it was a default many picked up. Whereas, these others people went to willingly even when POSIX was available with even more portability. That's why they were objectively more successful on portability.

Re:Posix Has Become Outdated (2016) [pdf] by AsyncAwait on Saturday February 11, 2017 @04:20PM (#13624865)
> It didn't achieve the popularity it sought

By what metric?

POSIX was never trying to be the universal interface for all kinds of applications, just some sane common interface for UNIX vendors, all of which do implement POSIX to a large extent these days.

You could perhaps make the argument that by the time POSIX saw enough adoption, software development evolved and that nowdays there are lots of interfaces that are not part of POSIX/not portable, but that's a slightly different issue.

Re:Posix Has Become Outdated (2016) [pdf] by nickpsecurity on Saturday February 11, 2017 @06:15PM (#13625439)
That's close to my argument. There's kind of two conversations going on in these threads: UNIX portability and general portability. POSIX is fine on former but worse on latter since it wasnt designed for it (among other things).
Re:Posix Has Become Outdated (2016) [pdf] by geezerjay on Saturday February 11, 2017 @12:07PM (#13623466)
> I would not if the goal of a standard is uptake by OS and application developers.

That point is moot, as those platforms were designed with the express purpose of being particularly hostile regarding interoperability while forcing other corporate products as alternatives, in a well-known lock-in strategy.

Re:Posix Has Become Outdated (2016) [pdf] by nickpsecurity on Saturday February 11, 2017 @12:32PM (#13623609)
"those platforms were designed with the express purpose"

They were designed with the purpose of giving users a way to get things done that was profitable to the company. The first part along with lock-in techniques are why most of the desktop market is Windows with a large chunk of the server market. That means the kind of portability that matters to people wanting to maximize benefit to users or profits to the company better include a Windows version. There are portability alternatives to POSIX that allow that usually with other benefits on top of it. Far from moot...

Re:Posix Has Become Outdated (2016) [pdf] by geezerjay on Saturday February 11, 2017 @12:50PM (#13623731)
> They were designed with the purpose of giving users

No, they actually weren't.

The "users" don't have any say whether Windows complies with any standard, or even if Microsoft breaks all of them to try to force the world to submit to their "vendor lock-in" business strategy.

Trying to pass off the consequences of vendor lock-in policies as technical arguments is somewhere between absurd and disingenuous.

Re:Posix Has Become Outdated (2016) [pdf] by nickpsecurity on Saturday February 11, 2017 @06:13PM (#13625431)
You said the only purpose of what Windows did was locking in people. I said it was to provide software that let people accomplish what they want while making profit sustained by lockin tricks. Are you saying Microsoft never intended people to use Windows API or software to accomplish their personal goals? That there is zero benefit to users of Windows in consumer or business space?
Re:Posix Has Become Outdated (2016) [pdf] by cperciva on Saturday February 11, 2017 @04:41AM (#13621938)
You (and the authors of the article) seem to be assuming that "modern applications" are by definition applications with graphical user interfaces.
Re:Posix Has Become Outdated (2016) [pdf] by kajecounterhack on Saturday February 11, 2017 @05:00AM (#13621982)
Colin I think you're being a little pedantic. You can use context to infer what they mean.

This is their application list:

https://github.com/columbia/libtrack/blob/master/workloads/o...

Re:Posix Has Become Outdated (2016) [pdf] by cperciva on Saturday February 11, 2017 @05:22AM (#13622032)
Looking at that list, I'm struck by the fact that it includes most of the applications users know they're using, but only a small minority of the processes running on an OS X system.
Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @05:26AM (#13622042)
> only a small minority of the processes running on an OS X system.

Which are anyway not POSIX.

https://www.amazon.de/MAC-OS-Internals-Amit-Singh/dp/0321278...

Re:Posix Has Become Outdated (2016) [pdf] by netheril96 on Saturday February 11, 2017 @07:39AM (#13622361)
POSIX is an application interface. How OS X is implemented under the hood is irrelevant. It does present an application interface mostly conformant to POSIX standards, so it is some way POSIX.
Re:Posix Has Become Outdated (2016) [pdf] by coldtea on Saturday February 11, 2017 @08:33AM (#13622513)
>so it is some way POSIX.

That's hardly encouraging, when all the basics and all the newer OS X abstractions are based things other than POSIX standards, and ask programmers to connect to them with non-POSIX interfaces.

The fact that it also supports a POSIX layer is about as relevant to what we're discussing (whether POSIX is fit for today's needs and used in today's mainstream development) as the presence of some X Windows translation layer is in Wayland.

Re:Posix Has Become Outdated (2016) [pdf] by kccqzy on Saturday February 11, 2017 @04:43AM (#13621942)
That's what majority of computer users (smartphones included) use though. As for developer-oriented CLI tools, I've also seen more and more GNU-isms and macOS-isms creeping in.
Re:Posix Has Become Outdated (2016) [pdf] by tmalsburg2 on Saturday February 11, 2017 @06:45AM (#13622208)
Counter example: Almost everyone uses the Apache webserver when they browse the web, but hardly anyone even knows what that is. We could easily compile a long list of similar crucial infrastructure software that virtually everyone is using and that doesn't have graphical user interfaces.
Re:Posix Has Become Outdated (2016) [pdf] by johannes1234321 on Saturday February 11, 2017 @09:50AM (#13622822)
Even if everybody is using Apache: Apache httpd doesn't use POSIX but APR (Apache Portable Runtime) which removes the POSIX-dependenc from the core of httpd.
Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @07:06AM (#13622258)
How much of Apache is pure POSIX and how much are #ifdef around OS specific APIs?
Re:Posix Has Become Outdated (2016) [pdf] by nickpsecurity on Saturday February 11, 2017 @11:19AM (#13623211)
Apache uses the Apache Portable Runtime:

https://en.wikipedia.org/wiki/Apache_Portable_Runtime

That's one of the competitors to POSIX for application portability. Unlike POSIX, it was totally open as well vs people telling me The Open Group will sell me a copy of full standard.

Re:Posix Has Become Outdated (2016) [pdf] by mikekchar on Saturday February 11, 2017 @10:03AM (#13622885)
Interestingly, I took a random tour through the source code and didn't find a single OS specific API. Which is not to say that the code was intrinsically POSIX specific. It is a surprisingly good example of multi-platform code. The #ifdefs that I saw were all to do with defining functionality from scratch if it didn't exist rather than the more usual massively indented list of platform specific hacks. Although I had fairly high expectations, I was fairly surprised at how nice the code was.

I didn't spend all that much time, so I'm sure I missed a lot, but I recommend taking a look for yourself as it is quite interesting.

Re:Posix Has Become Outdated (2016) [pdf] by ycmbntrthrwaway on Saturday February 11, 2017 @06:56AM (#13622236)
Count sendmail in:

> There is some sort of perverse pleasure in knowing that it's basically impossible to send a piece of hate mail through the Internet without its being touched by a gay program. That's kind of funny.

Re:Posix Has Become Outdated (2016) [pdf] by nj923f on Saturday February 11, 2017 @11:31AM (#13623272)
sendmail is dead at this point though.
Re:Posix Has Become Outdated (2016) [pdf] by ci5er on Saturday February 11, 2017 @02:03PM (#13624127)
How so? As recently as 2015, sendmail, Microsoft Exchange Server, Postfix, and Exim together represented over 90% of SMTP service.

Or is this one of those pre-emptive "called it first!" kind of comments?

Re:Posix Has Become Outdated (2016) [pdf] by yosamino on Saturday February 11, 2017 @02:34PM (#13624290)
I think the point is that out of these 90% not so many ones are "sendmail" anymore.
Re:Posix Has Become Outdated (2016) [pdf] by justinator on Saturday February 11, 2017 @12:37PM (#13623645)
As a suggestion, let's keep the homophobia to a nonexistent level, please.
Re:Posix Has Become Outdated (2016) [pdf] by detaro on Saturday February 11, 2017 @12:43PM (#13623690)
That's a quote by the (gay) developer of sendmail, I don't see how quoting that is homophobic?
Re:Posix Has Become Outdated (2016) [pdf] by Avshalom on Saturday February 11, 2017 @10:16AM (#13622957)
Apache almost certainly uses epoll/kqueue.
Re:Posix Has Become Outdated (2016) [pdf] by gm-conspiracy on Saturday February 11, 2017 @07:22AM (#13622303)
Almost everyone uses the Apache webserver when they browse the web

No.

https://news.netcraft.com/archives/category/web-server-surve...

Re:Posix Has Become Outdated (2016) [pdf] by tmalsburg2 on Saturday February 11, 2017 @07:31AM (#13622332)
Reading comprehension. I did not say that people only visit pages that are served by Apache. Even if Apache has just 20% market share there is a very high chance that you will hit some pages served by Apache. Plus, the same argument really applies to most alternatives to Apache.
Re:Posix Has Become Outdated (2016) [pdf] by yellowapple on Saturday February 11, 2017 @10:03AM (#13622883)
To be fair, Apache not having an overwhelmingly-majority market share is (according to that site) a relatively recent trend.
Re:Posix Has Become Outdated (2016) [pdf] by gkya on Saturday February 11, 2017 @09:37AM (#13622782)
Nitpicking 101.
Re:Posix Has Become Outdated (2016) [pdf] by geezerjay on Saturday February 11, 2017 @10:14AM (#13622937)
> That's what majority of computer users (smartphones included) use though.

Most computer users access countless sites which run on deep software stacks running on any flavour of linux.

...which run POSIX.

Re:Posix Has Become Outdated (2016) [pdf] by bluejekyll on Saturday February 11, 2017 @11:40AM (#13623327)
> ...which run POSIX.

More accurate to say, expose a POSIX compliant API.

I remember BeOS had a 100% compliant POSIX layer. Impressive for a non-UNIX.

Re:Posix Has Become Outdated (2016) [pdf] by yamaneko on Saturday February 11, 2017 @08:30AM (#13622503)
Just curious, can you give me some examples of GNU-isms/macOS-isms? I have a suspicion what they might be, but I'm not sure.
Re:Posix Has Become Outdated (2016) [pdf] by bluejekyll on Saturday February 11, 2017 @08:50AM (#13622599)
Asynchronous IO, hardly anyone uses the POSIX standard of select and poll anymore, and instead use kqueue (bsd/macOS) and epoll (Linux).

I should add that this is for performance critical software.

Re:Posix Has Become Outdated (2016) [pdf] by JdeBP on Saturday February 11, 2017 @10:15AM (#13622945)
I'll agree that a lot of people prefer and recommend things like kqueue+kevent over select/poll. But my experience is that the statement that "hardly anyone uses" select/poll is still erroneous.

On an out-of-the-box installation of FreeBSD/TrueOS, most processes listed by "ps" show a wchan of things like ttyin, select, poll, pause, uwait, wait, or nanslp. On the FreeBSD machine that I am typing at right now exactly one program is waiting in kqread, one of the inner processes of Chrome.

Install my nosh toolset, and a fair amount of that changes, because my programs use kqueue+kevent and suddenly the "ps" output is full of kqreads. But what's left is still not "hardly anyone", by a long chalk.

Would that it were! There's a nasty kernel panic somewhere in the innards of the kevent system on FreeBSD, where a sleeping thread holds a non-sleepable lock, that only manifests itself (at least for me) when there are a lot of programs using kqueue+kevent on a system that is doing a lot of stuff. If more people used kqueue+kevent in more programs, there would be more people wanting it fixed. (-:

Re:Posix Has Become Outdated (2016) [pdf] by bluejekyll on Saturday February 11, 2017 @11:31AM (#13623273)
Yeah. I considered this critique, which is why I added the performance caveat. The thing is, libraries of other languages are being written abstractly over the native eventing systems on the OSes.

For example, the MIO library in Rust, is an abstraction over epoll, kqueue, and the MS primative. Not one over the POSIX standard. This means that if you use that in a Rust library, you get that platform independence for free.

Re:Posix Has Become Outdated (2016) [pdf] by cperciva on Saturday February 11, 2017 @01:46PM (#13624038)
The standard select and poll syscalls perform just fine if you only have a handful of sockets. Heck, they perform just fine as long as a nontrivial proportion of your sockets are ready at any given time.

The place you need kqueue and epoll is when most of your connections are idle -- IRC servers and web servers with large keepalive times. But the vast majority of processes are not IRC or web servers.

Re:Posix Has Become Outdated (2016) [pdf] by jmgao on Saturday February 11, 2017 @05:10PM (#13625145)
One hilarious one is basename(3), which is two functions with different behavior, but the same name. The one you get depends on which headers you include, and what _GNU_SOURCE is set to.

See http://man7.org/linux/man-pages/man3/basename.3.html for the manpage.

Re:Posix Has Become Outdated (2016) [pdf] by astrange on Saturday February 11, 2017 @03:15PM (#13624536)
dispatch and XPC are very popular macOS APIs that are "POSIX level". At least one of those is open source.
Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @05:01AM (#13621984)
If I wanted to stay with CLI applications, MS-DOS would be enough.

UNIX got X Windows in 1987.

Re:Posix Has Become Outdated (2016) [pdf] by astrobe_ on Saturday February 11, 2017 @05:45AM (#13622079)
CLI and GUI have different grammars and capabilities. If you think that CLI means "prehistory" and GUI means "future", then ask yourself why CLI has "survived" GUI for at least 30 years.
Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @05:53AM (#13622099)
Has it?

Sure it is down there on the OS tooling, but how many people outside UNIX devs, do actually bother using it?

Personally it doesn't offer me any benefit over an higher level programming language REPL.

Re:Posix Has Become Outdated (2016) [pdf] by AnonymousPlanet on Saturday February 11, 2017 @07:19AM (#13622297)
At the end of the 80s and throughout the 90s, a common belief was that the CLI could be completely replaced by the GUI, one just hadn't yet figured out how. The GUI, however, never delivered those last 10-20% that you need in automation or in a context that requires reproducibility.

Windows NT was designed towards those promises, but eventually Microsoft realised their mistake and created - ten years ago - Powershell.

The entire dev world has been catching on to the fact that you get reproducible procedures when you use a CLI. Data scientists are another group who know how much power they get from a CLI.

Yet the myth of the obsolete CLI and the omnipotent GUI still is strong. Especially with those people who never really worked with a decent CLI.

Re:Posix Has Become Outdated (2016) [pdf] by JdeBP on Saturday February 11, 2017 @08:40AM (#13622542)
Microsoft didn't have a mistake to realize. From very early on, one of the must-have toolsets for Windows NT was the Windows NT Resource Kit, with a whole bunch of command-line tools -- provided by Microsoft. There was in fact a whole load of command-line tooling for Windows NT, well before PowerShell and back to version 3.1 .

Far from this being the reversal that you propound, actual history is even more in favour of your argument. (-:

Re:Posix Has Become Outdated (2016) [pdf] by AnonymousPlanet on Saturday February 11, 2017 @11:06AM (#13623157)
Look at the design of NT. Every output is based on IPC and calls to the event logger. That something like a stdout exists is due to backwards compatibility and mostly the Posix subsystem. The entire OS was meant to be used and administrated by means of GUI. The existence of a bunch of CLI tools that deviate from the main paradigm doesn't prove anything to the contrary. Have a look at the literature on the design and concepts of NT.
Re:Posix Has Become Outdated (2016) [pdf] by kyberias on Saturday February 11, 2017 @11:31AM (#13623274)
>"Every output is based on IPC and calls to the event logger."

What does that even mean?

I think you're painting an incorrect picture of the design goals of NT. The original NT was designed by a group of people (e.g. Cutler) with VMS background and they sure as hell didn't have any GUI in mind. In fact, Windows came to the picture much later. Whether NT is administered by a GUI or CLI is totally irrelevant to NT design. Today all the NT derivatives (Windows Server) can be administered with a rich variety of command line tools.

Stdout exists because the original subsystems (OS/2, Windows, MS-DOS) required that. Same applies TODAY. Where is the NT bias to GUIs exactly?

Re:Posix Has Become Outdated (2016) [pdf] by dfox on Saturday February 11, 2017 @03:40PM (#13624679)
To be fair, original Windows API didn't have any concept of stdin/out/err. In win16 days most development environments had something like that but it was emulated inside compiler vendor supplied runtime libraries. CLI as first class OS APIs come from NT.
Re:Posix Has Become Outdated (2016) [pdf] by coldtea on Saturday February 11, 2017 @08:36AM (#13622526)
>At the end of the 80s and throughout the 90s, a common belief was that the CLI could be completely replaced by the GUI

And it has, except for backend programmers. Most Windows programmers stay all day in Visual Studio and the like, for example.

But as far as end users as concerned, that is the 99% of computer users, the CLI might as well not exist.

Re:Posix Has Become Outdated (2016) [pdf] by bluejekyll on Saturday February 11, 2017 @09:06AM (#13622652)
Ok. I'll bite. You're speaking from the perspective of someone locked into a way of doing work through the GUI, limiting your development experience to that of what the GUI developer (MS in this case) has provided you.

The CLI offers an infinitely combinable and customizable development experience, where huge ammounts of innovation have been occuring. You can see this in modern web development toolchains and new build tools for new languages, which simply would not happen if you restrict your Dev experience to just the GUI. I keep going back to just creating Makefiles for simple automation tasks.

Even the GUIs are starting to learn from this, which is why I think VSCode and Atom are becoming popular; they make it very simple to integrate new CLI tools into the GUIs workflow.

Re:Posix Has Become Outdated (2016) [pdf] by coldtea on Saturday February 11, 2017 @06:24PM (#13625479)
>Ok. I'll bite. You're speaking from the perspective of someone locked into a way of doing work through the GUI, limiting your development experience to that of what the GUI developer (MS in this case) has provided you.

No, I'm speaking from the perspective of someone that has worked in the old days with Sun OS (pre-Solaris) workstations and HP-UX machines, has used (for years) actual VT102 terminals, started using Linux distros around '97 and has been using the CLI in various forms since the mid-eighties or so.

There's nothing limited about the GUI (in theory, and, for most things that matter, in practice too). For example a visual flow language (think Automator in OS X or Quartz Composer, etc) can achieve all the "configurable pipeline" stuff people like in the CLI in a more controlled and formal way.

I also dislike the "dynamic typing" (everything is text/bytes) in the CLI, and would prefer something like PowerShell to have been the norm (with the accompanying toolset). Also note that there's nothing about the GUI that prevents someone from typing some commands too -- and in fact that's part of how lots of GUIs work.

Also, I wouldn't point to "modern web development toolchains" as something to advertise CLIs -- what would that be, Gulp, Grunt, Webpack and the like? All crufty solutions to inadequacies in steering the underlying language properly.

Even in theory, CLIs are not inherently more powerful, it's the other way around (they lack a dimension that GUIs offer (graphics) -- GUIs on the other hand don't lack any dimension -- they can incorporate text commands and text fields and pipelines just fine.

Re:Posix Has Become Outdated (2016) [pdf] by AnonymousPlanet on Saturday February 11, 2017 @11:13AM (#13623187)
Most Windows programmers I know have never really used an environment that doesn't punish you for using the command line. So what's your point? That most people you know never realised that many of their daily tasks can be easily automated if they wouldn't think that endless, tedious clickfests are the norm, and that this somehow proves that the CLI is obsolete?
Re:Posix Has Become Outdated (2016) [pdf] by coldtea on Saturday February 11, 2017 @06:24PM (#13625484)
>Most Windows programmers I know have never really used an environment that doesn't punish you for using the command line. So what's your point?

That they don't miss it either, and that the "endless, tedious clickfests" are a myth of people who don't know the other side.

Re:Posix Has Become Outdated (2016) [pdf] by AnonymousPlanet on Sunday February 12, 2017 @05:46AM (#13627784)
It's not a myth if you watch it happening nearly every day.
Re:Posix Has Become Outdated (2016) [pdf] by JdeBP on Saturday February 11, 2017 @09:38AM (#13622784)
I doubt that there is anyone at all in this entire discussion that can state with authority what "most Windows programmers" do. We can point to what we and the people we have met and talked to about this do, but to go from that comapratively scant few handsful of people to a statement the size of "most windows programmers" is an overgeneralization by several orders of magnitude.
Re:Posix Has Become Outdated (2016) [pdf] by coldtea on Saturday February 11, 2017 @06:30PM (#13625509)
I doubt that discussion and insight can happen without generalising. I also doubt that we need to cover 100% or even 99.999% of Windows programmers to be able to speak about what "most Windows programmers do".

Anybody who has experience from 2-5 different Windows shops and been to a few relevant conferences can make a quite accurate assessment of what "most Windows programmers" do.

Re:Posix Has Become Outdated (2016) [pdf] by umanwizard on Saturday February 11, 2017 @12:33PM (#13623613)
I write client-side code in Xcode but I use the terminal all day for various things.
Re:Posix Has Become Outdated (2016) [pdf] by dualogy on Saturday February 11, 2017 @06:29AM (#13622173)
Silly flamefest this. Even back when I worked mostly for Windows/MS shops or clients, writing CLI tooling for business (not developer) needs were often the order of the day. Batch import/export tools and what not. Also prototypes of daemon-like Windows Services for client testing. All kinds of tools that weren't directly developer tools. Sure enough, we were interfacing with IT folks at the client end but still, no programmers per se!

There's a reason GUI starts with a G. Other than "graphical" user interfaces, "other" user interfaces will also always have their place..

In some ways looking at how much non-developer computer usage in the past decades has been in front of excel/access sheets and tables, these aren't so "graphical" either. They essentially end up evolving into (haphazard) custom CLI-like keyboard-driven power tools, half-programmable, half requiring intimate familiarity with its "commands" (formulas) or even "training" to use them fluidly.

But if you'd rather code out your CLI needs in a PL REPL (aka CLI), more power to you ;D

Re:Posix Has Become Outdated (2016) [pdf] by lomnakkus on Saturday February 11, 2017 @07:31AM (#13622331)
Yes, it has.

I mean there may not be as many users, but they are massively more powerful (in the realm of data processing only, alas) than people who don't know the CLI. It may be obvious to us, but I should perhaps also point out that CLI ~= REPL for the language that you're using (Bash or whatever).

Sure, these days CLI may mean "a Jupyter notebook", but the same principle applies.

It may not always look the same, but the point is that a CLI/REPL should be a combinatorial force multiplier. Otherwise, it's not really worthwhile.

Re:Posix Has Become Outdated (2016) [pdf] by Gravityloss on Saturday February 11, 2017 @06:51AM (#13622222)
Interesting, I assume you do development? How do you handle scripting, moving files, grepping logs etc?
Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @07:07AM (#13622262)
GUI tooling? Languages like Python?
Re:Posix Has Become Outdated (2016) [pdf] by umanwizard on Saturday February 11, 2017 @12:43PM (#13623691)
How would you do this with GUI tooling:

Find all files containing a particular string in a directory tree. For each of those files, if its filename appears in a file called Whitelist, write it to a file called Output. Otherwise, do nothing.

Here is one possible command line version (I haven't tested it as I'm on mobile):

TMPF=`mktemp /tmp/XXXXXXXX` && ag MyString | awk -F: '{print $1}' | sort | uniq > $TMPF && comm -12 Whitelist $TMPF > Output && rm $TMPF

Re:Posix Has Become Outdated (2016) [pdf] by hawski on Saturday February 11, 2017 @01:21PM (#13623882)
You probably could use process substitution to get rid of the temp file:

  comm -12 Whitelist <(ag MyString | awk -F: '{print $1}' | sort | uniq) > Output
Then you could use unique flag of sort:

  comm -12 Whitelist <(ag MyString | awk -F: '{print $1}' | sort -u) > Output
Also ag can list only matching files (getting rid of awk and unique):

  comm -12 Whitelist <(ag -l MyString | sort) > Output
Re:Posix Has Become Outdated (2016) [pdf] by umanwizard on Saturday February 11, 2017 @01:48PM (#13624054)
Thanks for the tips!
Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @03:25PM (#13624600)
Apparently you forgot to read the part where I mentioned "Languages like Python?".

    import glob
    import shutil
     
    whitelist =  set(open('Whitelist').read().split())
    for fd in [ x for x in glob.glob('mydir/*', recursive=True) if x in whitelist]:
       shutil.copy(fd, 'OUTPUT')

Better, no need to create temporary files.
Re:Posix Has Become Outdated (2016) [pdf] by belltaco on Saturday February 11, 2017 @02:33PM (#13624289)
Would be trivial to write a GUI program to do the same.
Re:Posix Has Become Outdated (2016) [pdf] by umanwizard on Saturday February 11, 2017 @02:52PM (#13624395)
I think it would take at least a half hour, unless you have directory traversal, file I/O, and regex APIs memorized which most people don't. That's "trivial" in the grand scheme of things but it's really not practical to do day-in and day-out every time some random scripting necessity arises.

And you presumably have to clutter up your filesystem with random project files, etc, which is annoying. Are you going to have 1000 of these, one for every time you need to do some sort of minor task like this?

That command, on the other hand, took me 2 or 3 minutes to write, and I had to look up the syntax for `mktemp`. If I had remembered the mktemp syntax it would have taken a few seconds.

Re:Posix Has Become Outdated (2016) [pdf] by gkya on Saturday February 11, 2017 @09:35AM (#13622772)
What? Those are not even POSIX environments (well, their primary and intended interface interface is not POSIX, but they maybe eventually inherit some of it because of their underlying environment or Unix influences, but that's irrelevant). Also, I never used OS X but AFAIK many Unix software runs on it easily, so it must be mostly compliant if not completely (IIRC it's a certified Unix).
Re:Posix Has Become Outdated (2016) [pdf] by johannes1234321 on Saturday February 11, 2017 @09:46AM (#13622813)
That's exactly the point of the report: Hardly anybody uses POSIX. Many people use frameworks on top of it though. If POSIX would go away many won't notice.
Re:Posix Has Become Outdated (2016) [pdf] by gkya on Saturday February 11, 2017 @09:59AM (#13622861)
Isn't a nice common interface that those frameworks and higher-level language libraries and interpreters can be written for a nice thing? It's like a nice road network, you sure want to be in your car not on the road, but it's better if most of it is good for the common tire, you don't want to change your tires at every turn.
Re:Posix Has Become Outdated (2016) [pdf] by nickpsecurity on Saturday February 11, 2017 @11:11AM (#13623178)
It's a nice thing if you're trying to run on many UNIX's. Most projects just target one these days: Linux. If it's mobile supported, then add Android which is also a Linux. That means they'd only have to code the portability framework for 1-2 POSIX-like OS's. POSIX offers about no advantage there. They'll just wrap it like they did Windows and Mac.
Re:Posix Has Become Outdated (2016) [pdf] by johannes1234321 on Saturday February 11, 2017 @10:55AM (#13623115)
Yes, but the need for those shows that POSIX failed.
Re:Posix Has Become Outdated (2016) [pdf] by AsyncAwait on Saturday February 11, 2017 @04:29PM (#13624918)
POSIX mostly wasn't even designed to handle the things most of these frameworks do, so how it could be said that it failed?
Re:Posix Has Become Outdated (2016) [pdf] by gkya on Saturday February 11, 2017 @10:58AM (#13623131)
Just like CPU opcodes have failed... POSIX is a low level interface, that of the OS and the C library. It's another level of abstraction, necessary for the ones that are building other levels on top of it.
Re:Posix Has Become Outdated (2016) [pdf] by johannes1234321 on Saturday February 11, 2017 @11:34AM (#13623291)
CPU opcodes have a different purpose. POSIX aims to be an API for general purpose application development.
Re:Posix Has Become Outdated (2016) [pdf] by AsyncAwait on Saturday February 11, 2017 @04:31PM (#13624933)
...for UNIX systems, where we're talking about mostly CLI apps, since POSIX was released a decade before the iPhone.

I'd say if you can run bash and gcc on Linux, OSX and BSD, it mostly succeeded.

Re:Posix Has Become Outdated (2016) [pdf] by AsyncAwait on Saturday February 11, 2017 @04:27PM (#13624907)
I don't think that's true. POSIX was written in the 90s, when many of these frameworks didn't exist. It's still nice that I can run GNU tools on basically any POSIX platform and I think many (developers) would notice its disappearance.

User's probably won't notice, but they tend not to care about standards anyway.

Re:Posix Has Become Outdated (2016) [pdf] by tedunangst on Saturday February 11, 2017 @09:17AM (#13622694)
Alright, so when did posix become outdated? Was it outdated 20 years ago because you couldn't write a Symbian app with standard posix interfaces?
Re:Posix Has Become Outdated (2016) [pdf] by quickben on Saturday February 11, 2017 @08:34AM (#13622517)
Arguably, you listed the projects that went out of their way heavily to restrict and lock-in users.
Re:Posix Has Become Outdated (2016) [pdf] by bluejekyll on Saturday February 11, 2017 @08:46AM (#13622570)
Isn't the point that those are the most prolific platforms, where a common API to the system, POSIX, would be most beneficial to us as developers? But it's simply not possible to do?

From what I can tell POSIX has been replaced by per language stdlibs, with the portability of your software tied to the portability of a languages stdlib.

POSIX was mainly designed for C, most other languages have a higher level abstraction.

Re:Posix Has Become Outdated (2016) [pdf] by JdeBP on Saturday February 11, 2017 @09:47AM (#13622818)
POSIX wasn't designed, per se. It's a standard, a lot of which was the codification of existing practices on operating systems where the C language was the primary language.

Also note the existence of FORTRAN language bindings, and the low level but persistent rumbles about C++ language bindings that have continued for the past 20 years or so.

Then note the number of languages where the doco tells you all about how "we wrap the C library", but what is actually there is in effect a language binding for POSIX for that language.

Re:Posix Has Become Outdated (2016) [pdf] by nickpsecurity on Saturday February 11, 2017 @11:15AM (#13623196)
"POSIX wasn't designed, per se. It's a standard, a lot of which was the codification of existing practices on operating systems where the C language was the primary language."

That's what I was going to say about UNIX, C, and POSIX. The UNIX and C languages' made tradeoffs specifically oriented around the weak hardware they had. We no longer need those. POSIX evolved from a bunch of competing vendors doing a mix of justifiable and weird stuff with a standard trying to abstract it all. It's inherently going to be worse than a clean design over the smaller number of modern UNIXen without all the politics that went into POSIX.

And then we have fact that many portable apps need to run on Windows, Mac, iPhone, etc POSIX ranged from useless to lacking platform-specific capabilities on these. So, a clean-slate design covering each of these platforms is the obvious solution. Fortunately, we have a bunch of them with varying styles and tradeoffs to choose from. :)

Re:Posix Has Become Outdated (2016) [pdf] by bluejekyll on Saturday February 11, 2017 @09:55AM (#13622847)
Yes, you are completely correct. And 'designed' was perhaps a poor word choice.

In terms of wrappers of POSIX, I'm definitely not saying that POSIX is dead and not used. It's great that it is a beginning point for porting to various platforms. What I was more pointing out is that as a developer, its your stdlib's portability that is key. For example, many languages use POSIX interfaces where they can, and then switch to platform specific interfaces where necessary, for performance or other reasons.

Re:Posix Has Become Outdated (2016) [pdf] by geezerjay on Saturday February 11, 2017 @10:11AM (#13622923)
> Isn't the point that those are the most prolific platforms

You're confusing popularity with being technically antiquated or obsolete.

Re:Posix Has Become Outdated (2016) [pdf] by bluejekyll on Saturday February 11, 2017 @11:36AM (#13623308)
I think your reading more into my comment than is there.

If you're trying to write a portable app between all of those platforms, would you choose POSIX or would you instead pick a framework or language that has a better high level abstraction?

Re:Posix Has Become Outdated (2016) [pdf] by geezerjay on Saturday February 11, 2017 @11:59AM (#13623416)
Again, you're confusing popularity with being technically antiquated or obsolete.

The thesis presented in the article is that POSIX is outdated. The popularity of a platform, particularly one which was forced upon the world through a de-facto monopoly, is entirely irrelevant in any argument on the technical merits of a technical standard, particularly when some of those platforms were designed with the express purpose of locking out virtually all standardized programming languages.

Re:Posix Has Become Outdated (2016) [pdf] by bluejekyll on Saturday February 11, 2017 @12:45PM (#13623698)
Hm. The point of POSIX is to support cross platform development with a single standard. If you find that it is not keeping pace with the most modern and popular platforms, is it not outdated?

Technically speaking I think the standard is great, especially around having consistent definitions for thing like close(). But there are better options on all platforms to the select() and poll() interfaces.

In terms of my desires for open platforms that support open standards, I'm completely with you. If anything what we should be debating is how to update the POSIX standard such that it is worthwhile to target during development.

I don't see how it's helpful to ignore what is popular out in the world.

Re:Posix Has Become Outdated (2016) [pdf] by geezerjay on Saturday February 11, 2017 @12:57PM (#13623765)
> Hm. The point of POSIX is to support cross platform development with a single standard.

No, the point of POSIX is to specify a set of interfaces that software vendors should provide and target if their goal is to provide an UNIX variant or develop UNIX-compliant software.

The key issue is that the main requirement is willingness on behalf of the software vendors to comply with the standard, and offer/target the standardized interface.

If a software company is blatantly hostile regarding interoperability, and bases their business model on vendor lock-in strategies then it's quite obvious that those companies are particularly are not willing to play ball with others regarding interoperability.

Yet, somehow you're presenting those companies, and the consequences of the actions taken by companies that are openly hostile regarding interoperability, as some sort of proof that a specific interoperability target suffers from technical problems.

Re:Posix Has Become Outdated (2016) [pdf] by ajross on Saturday February 11, 2017 @10:23AM (#13622990)
It's the other way around, really. The OS is what provides the POSIX API to the higher levels of the stack. All of those are POSIX under the hood, and their middleware is dominated by POSIX code that is reasonably portable.
Re:Posix Has Become Outdated (2016) [pdf] by tinus_hn on Monday February 13, 2017 @07:00AM (#13634647)
Gee I wonder why you cannot write a program for an interface that nobody imagined would be possible in 2000, using only a standard described in the late 80s. Must be some kind of conspiracy..
Re:Posix Has Become Outdated (2016) [pdf] by tr1ck5t3r on Saturday February 11, 2017 @05:39AM (#13622066)
Totally agree, the authors seem to be missing the point that these libraries are a new layer between the posix api's and the end result which is typically a GUI application. The fact these libraries also bring a walled garden to the table which bypass the posix api's also demonstrate how security is being bypassed. Plus with the analysis only featuring the 3 os's Android, OS X and Ubuntu, they would have noticed that Windows OS's pretty much mirror what POSIX does as well.

1 "usage is driven by high-level frameworks, which impacts POSIX’s portability goals.", that's the libraries/frameworks fault, plus have they heard of GCC? It works on many OS's.

2 "extension APIs, namely ioctl, dominate modern POSIX usage patterns as OS developers increasingly use them to build support for abstractions missing from the POSIX standard" Perhaps they have missed the point, that without the POSIX standard these extensions would not exist in the first place. They might just complain about procedural coding practices being outdated when comparing to OOP coding practices, even though OOP brings slow bloat to the table as well. Perhaps a better question to ask is, why has a framework/library got some functionality which could have used some of the POSIX api's but didn't? Was it expedient to keep it within the framework/library and bypass the Posix API's altogether?

3."new abstractions are arising, driven by the same POSIX limitations across the three OSes, but the new abstractions are not converging." So abstractions are a new name for libraries and framework and not the concept like OOP coding is an abstraction. Perhaps complaining to the authors of libraries and frameworks that don't port would be better? POSIX is like a foundation that allows creativity to take place. This is why we see the differences in libs/frameworks, different OS's have different parameters, different design ethos or objectives. Even in Windows, some API's are used more frequently than others, for example I cant remember when I last used the API's involved in getting a dial up modem to work, or using API's to print direct to a dot matrix printer. This paper just seems to be an analysis of API's not used as frequently as they used to be, sometimes because libraries/frameworks have rightly or wrongly replaced them for a myriad of reasons.

Qudos to them for downloading so many apps and analysing them though. Just what you need if you are looking for backdoors en-masse or exploitable bugs.