News for hackers. Stuff that matters.
Re:Posix Has Become Outdated (2016) [pdf] by cperciva on Saturday February 11, 2017 @04:41AM (#13621938)
You (and the authors of the article) seem to be assuming that "modern applications" are by definition applications with graphical user interfaces.
Re:Posix Has Become Outdated (2016) [pdf] by kajecounterhack on Saturday February 11, 2017 @05:00AM (#13621982)
Colin I think you're being a little pedantic. You can use context to infer what they mean.

This is their application list:

https://github.com/columbia/libtrack/blob/master/workloads/o...

Re:Posix Has Become Outdated (2016) [pdf] by cperciva on Saturday February 11, 2017 @05:22AM (#13622032)
Looking at that list, I'm struck by the fact that it includes most of the applications users know they're using, but only a small minority of the processes running on an OS X system.
Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @05:26AM (#13622042)
> only a small minority of the processes running on an OS X system.

Which are anyway not POSIX.

https://www.amazon.de/MAC-OS-Internals-Amit-Singh/dp/0321278...

Re:Posix Has Become Outdated (2016) [pdf] by netheril96 on Saturday February 11, 2017 @07:39AM (#13622361)
POSIX is an application interface. How OS X is implemented under the hood is irrelevant. It does present an application interface mostly conformant to POSIX standards, so it is some way POSIX.
Re:Posix Has Become Outdated (2016) [pdf] by coldtea on Saturday February 11, 2017 @08:33AM (#13622513)
>so it is some way POSIX.

That's hardly encouraging, when all the basics and all the newer OS X abstractions are based things other than POSIX standards, and ask programmers to connect to them with non-POSIX interfaces.

The fact that it also supports a POSIX layer is about as relevant to what we're discussing (whether POSIX is fit for today's needs and used in today's mainstream development) as the presence of some X Windows translation layer is in Wayland.

Re:Posix Has Become Outdated (2016) [pdf] by kccqzy on Saturday February 11, 2017 @04:43AM (#13621942)
That's what majority of computer users (smartphones included) use though. As for developer-oriented CLI tools, I've also seen more and more GNU-isms and macOS-isms creeping in.
Re:Posix Has Become Outdated (2016) [pdf] by tmalsburg2 on Saturday February 11, 2017 @06:45AM (#13622208)
Counter example: Almost everyone uses the Apache webserver when they browse the web, but hardly anyone even knows what that is. We could easily compile a long list of similar crucial infrastructure software that virtually everyone is using and that doesn't have graphical user interfaces.
Re:Posix Has Become Outdated (2016) [pdf] by johannes1234321 on Saturday February 11, 2017 @09:50AM (#13622822)
Even if everybody is using Apache: Apache httpd doesn't use POSIX but APR (Apache Portable Runtime) which removes the POSIX-dependenc from the core of httpd.
Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @07:06AM (#13622258)
How much of Apache is pure POSIX and how much are #ifdef around OS specific APIs?
Re:Posix Has Become Outdated (2016) [pdf] by nickpsecurity on Saturday February 11, 2017 @11:19AM (#13623211)
Apache uses the Apache Portable Runtime:

https://en.wikipedia.org/wiki/Apache_Portable_Runtime

That's one of the competitors to POSIX for application portability. Unlike POSIX, it was totally open as well vs people telling me The Open Group will sell me a copy of full standard.

Re:Posix Has Become Outdated (2016) [pdf] by mikekchar on Saturday February 11, 2017 @10:03AM (#13622885)
Interestingly, I took a random tour through the source code and didn't find a single OS specific API. Which is not to say that the code was intrinsically POSIX specific. It is a surprisingly good example of multi-platform code. The #ifdefs that I saw were all to do with defining functionality from scratch if it didn't exist rather than the more usual massively indented list of platform specific hacks. Although I had fairly high expectations, I was fairly surprised at how nice the code was.

I didn't spend all that much time, so I'm sure I missed a lot, but I recommend taking a look for yourself as it is quite interesting.

Re:Posix Has Become Outdated (2016) [pdf] by ycmbntrthrwaway on Saturday February 11, 2017 @06:56AM (#13622236)
Count sendmail in:

> There is some sort of perverse pleasure in knowing that it's basically impossible to send a piece of hate mail through the Internet without its being touched by a gay program. That's kind of funny.

Re:Posix Has Become Outdated (2016) [pdf] by nj923f on Saturday February 11, 2017 @11:31AM (#13623272)
sendmail is dead at this point though.
Re:Posix Has Become Outdated (2016) [pdf] by ci5er on Saturday February 11, 2017 @02:03PM (#13624127)
How so? As recently as 2015, sendmail, Microsoft Exchange Server, Postfix, and Exim together represented over 90% of SMTP service.

Or is this one of those pre-emptive "called it first!" kind of comments?

Re:Posix Has Become Outdated (2016) [pdf] by yosamino on Saturday February 11, 2017 @02:34PM (#13624290)
I think the point is that out of these 90% not so many ones are "sendmail" anymore.
Re:Posix Has Become Outdated (2016) [pdf] by justinator on Saturday February 11, 2017 @12:37PM (#13623645)
As a suggestion, let's keep the homophobia to a nonexistent level, please.
Re:Posix Has Become Outdated (2016) [pdf] by detaro on Saturday February 11, 2017 @12:43PM (#13623690)
That's a quote by the (gay) developer of sendmail, I don't see how quoting that is homophobic?
Re:Posix Has Become Outdated (2016) [pdf] by Avshalom on Saturday February 11, 2017 @10:16AM (#13622957)
Apache almost certainly uses epoll/kqueue.
Re:Posix Has Become Outdated (2016) [pdf] by gm-conspiracy on Saturday February 11, 2017 @07:22AM (#13622303)
Almost everyone uses the Apache webserver when they browse the web

No.

https://news.netcraft.com/archives/category/web-server-surve...

Re:Posix Has Become Outdated (2016) [pdf] by tmalsburg2 on Saturday February 11, 2017 @07:31AM (#13622332)
Reading comprehension. I did not say that people only visit pages that are served by Apache. Even if Apache has just 20% market share there is a very high chance that you will hit some pages served by Apache. Plus, the same argument really applies to most alternatives to Apache.
Re:Posix Has Become Outdated (2016) [pdf] by yellowapple on Saturday February 11, 2017 @10:03AM (#13622883)
To be fair, Apache not having an overwhelmingly-majority market share is (according to that site) a relatively recent trend.
Re:Posix Has Become Outdated (2016) [pdf] by gkya on Saturday February 11, 2017 @09:37AM (#13622782)
Nitpicking 101.
Re:Posix Has Become Outdated (2016) [pdf] by geezerjay on Saturday February 11, 2017 @10:14AM (#13622937)
> That's what majority of computer users (smartphones included) use though.

Most computer users access countless sites which run on deep software stacks running on any flavour of linux.

...which run POSIX.

Re:Posix Has Become Outdated (2016) [pdf] by bluejekyll on Saturday February 11, 2017 @11:40AM (#13623327)
> ...which run POSIX.

More accurate to say, expose a POSIX compliant API.

I remember BeOS had a 100% compliant POSIX layer. Impressive for a non-UNIX.

Re:Posix Has Become Outdated (2016) [pdf] by yamaneko on Saturday February 11, 2017 @08:30AM (#13622503)
Just curious, can you give me some examples of GNU-isms/macOS-isms? I have a suspicion what they might be, but I'm not sure.
Re:Posix Has Become Outdated (2016) [pdf] by bluejekyll on Saturday February 11, 2017 @08:50AM (#13622599)
Asynchronous IO, hardly anyone uses the POSIX standard of select and poll anymore, and instead use kqueue (bsd/macOS) and epoll (Linux).

I should add that this is for performance critical software.

Re:Posix Has Become Outdated (2016) [pdf] by JdeBP on Saturday February 11, 2017 @10:15AM (#13622945)
I'll agree that a lot of people prefer and recommend things like kqueue+kevent over select/poll. But my experience is that the statement that "hardly anyone uses" select/poll is still erroneous.

On an out-of-the-box installation of FreeBSD/TrueOS, most processes listed by "ps" show a wchan of things like ttyin, select, poll, pause, uwait, wait, or nanslp. On the FreeBSD machine that I am typing at right now exactly one program is waiting in kqread, one of the inner processes of Chrome.

Install my nosh toolset, and a fair amount of that changes, because my programs use kqueue+kevent and suddenly the "ps" output is full of kqreads. But what's left is still not "hardly anyone", by a long chalk.

Would that it were! There's a nasty kernel panic somewhere in the innards of the kevent system on FreeBSD, where a sleeping thread holds a non-sleepable lock, that only manifests itself (at least for me) when there are a lot of programs using kqueue+kevent on a system that is doing a lot of stuff. If more people used kqueue+kevent in more programs, there would be more people wanting it fixed. (-:

Re:Posix Has Become Outdated (2016) [pdf] by bluejekyll on Saturday February 11, 2017 @11:31AM (#13623273)
Yeah. I considered this critique, which is why I added the performance caveat. The thing is, libraries of other languages are being written abstractly over the native eventing systems on the OSes.

For example, the MIO library in Rust, is an abstraction over epoll, kqueue, and the MS primative. Not one over the POSIX standard. This means that if you use that in a Rust library, you get that platform independence for free.

Re:Posix Has Become Outdated (2016) [pdf] by cperciva on Saturday February 11, 2017 @01:46PM (#13624038)
The standard select and poll syscalls perform just fine if you only have a handful of sockets. Heck, they perform just fine as long as a nontrivial proportion of your sockets are ready at any given time.

The place you need kqueue and epoll is when most of your connections are idle -- IRC servers and web servers with large keepalive times. But the vast majority of processes are not IRC or web servers.

Re:Posix Has Become Outdated (2016) [pdf] by jmgao on Saturday February 11, 2017 @05:10PM (#13625145)
One hilarious one is basename(3), which is two functions with different behavior, but the same name. The one you get depends on which headers you include, and what _GNU_SOURCE is set to.

See http://man7.org/linux/man-pages/man3/basename.3.html for the manpage.

Re:Posix Has Become Outdated (2016) [pdf] by astrange on Saturday February 11, 2017 @03:15PM (#13624536)
dispatch and XPC are very popular macOS APIs that are "POSIX level". At least one of those is open source.
Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @05:01AM (#13621984)
If I wanted to stay with CLI applications, MS-DOS would be enough.

UNIX got X Windows in 1987.

Re:Posix Has Become Outdated (2016) [pdf] by astrobe_ on Saturday February 11, 2017 @05:45AM (#13622079)
CLI and GUI have different grammars and capabilities. If you think that CLI means "prehistory" and GUI means "future", then ask yourself why CLI has "survived" GUI for at least 30 years.
Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @05:53AM (#13622099)
Has it?

Sure it is down there on the OS tooling, but how many people outside UNIX devs, do actually bother using it?

Personally it doesn't offer me any benefit over an higher level programming language REPL.

Re:Posix Has Become Outdated (2016) [pdf] by AnonymousPlanet on Saturday February 11, 2017 @07:19AM (#13622297)
At the end of the 80s and throughout the 90s, a common belief was that the CLI could be completely replaced by the GUI, one just hadn't yet figured out how. The GUI, however, never delivered those last 10-20% that you need in automation or in a context that requires reproducibility.

Windows NT was designed towards those promises, but eventually Microsoft realised their mistake and created - ten years ago - Powershell.

The entire dev world has been catching on to the fact that you get reproducible procedures when you use a CLI. Data scientists are another group who know how much power they get from a CLI.

Yet the myth of the obsolete CLI and the omnipotent GUI still is strong. Especially with those people who never really worked with a decent CLI.

Re:Posix Has Become Outdated (2016) [pdf] by JdeBP on Saturday February 11, 2017 @08:40AM (#13622542)
Microsoft didn't have a mistake to realize. From very early on, one of the must-have toolsets for Windows NT was the Windows NT Resource Kit, with a whole bunch of command-line tools -- provided by Microsoft. There was in fact a whole load of command-line tooling for Windows NT, well before PowerShell and back to version 3.1 .

Far from this being the reversal that you propound, actual history is even more in favour of your argument. (-:

Re:Posix Has Become Outdated (2016) [pdf] by AnonymousPlanet on Saturday February 11, 2017 @11:06AM (#13623157)
Look at the design of NT. Every output is based on IPC and calls to the event logger. That something like a stdout exists is due to backwards compatibility and mostly the Posix subsystem. The entire OS was meant to be used and administrated by means of GUI. The existence of a bunch of CLI tools that deviate from the main paradigm doesn't prove anything to the contrary. Have a look at the literature on the design and concepts of NT.
Re:Posix Has Become Outdated (2016) [pdf] by kyberias on Saturday February 11, 2017 @11:31AM (#13623274)
>"Every output is based on IPC and calls to the event logger."

What does that even mean?

I think you're painting an incorrect picture of the design goals of NT. The original NT was designed by a group of people (e.g. Cutler) with VMS background and they sure as hell didn't have any GUI in mind. In fact, Windows came to the picture much later. Whether NT is administered by a GUI or CLI is totally irrelevant to NT design. Today all the NT derivatives (Windows Server) can be administered with a rich variety of command line tools.

Stdout exists because the original subsystems (OS/2, Windows, MS-DOS) required that. Same applies TODAY. Where is the NT bias to GUIs exactly?

Re:Posix Has Become Outdated (2016) [pdf] by dfox on Saturday February 11, 2017 @03:40PM (#13624679)
To be fair, original Windows API didn't have any concept of stdin/out/err. In win16 days most development environments had something like that but it was emulated inside compiler vendor supplied runtime libraries. CLI as first class OS APIs come from NT.
Re:Posix Has Become Outdated (2016) [pdf] by coldtea on Saturday February 11, 2017 @08:36AM (#13622526)
>At the end of the 80s and throughout the 90s, a common belief was that the CLI could be completely replaced by the GUI

And it has, except for backend programmers. Most Windows programmers stay all day in Visual Studio and the like, for example.

But as far as end users as concerned, that is the 99% of computer users, the CLI might as well not exist.

Re:Posix Has Become Outdated (2016) [pdf] by bluejekyll on Saturday February 11, 2017 @09:06AM (#13622652)
Ok. I'll bite. You're speaking from the perspective of someone locked into a way of doing work through the GUI, limiting your development experience to that of what the GUI developer (MS in this case) has provided you.

The CLI offers an infinitely combinable and customizable development experience, where huge ammounts of innovation have been occuring. You can see this in modern web development toolchains and new build tools for new languages, which simply would not happen if you restrict your Dev experience to just the GUI. I keep going back to just creating Makefiles for simple automation tasks.

Even the GUIs are starting to learn from this, which is why I think VSCode and Atom are becoming popular; they make it very simple to integrate new CLI tools into the GUIs workflow.

Re:Posix Has Become Outdated (2016) [pdf] by coldtea on Saturday February 11, 2017 @06:24PM (#13625479)
>Ok. I'll bite. You're speaking from the perspective of someone locked into a way of doing work through the GUI, limiting your development experience to that of what the GUI developer (MS in this case) has provided you.

No, I'm speaking from the perspective of someone that has worked in the old days with Sun OS (pre-Solaris) workstations and HP-UX machines, has used (for years) actual VT102 terminals, started using Linux distros around '97 and has been using the CLI in various forms since the mid-eighties or so.

There's nothing limited about the GUI (in theory, and, for most things that matter, in practice too). For example a visual flow language (think Automator in OS X or Quartz Composer, etc) can achieve all the "configurable pipeline" stuff people like in the CLI in a more controlled and formal way.

I also dislike the "dynamic typing" (everything is text/bytes) in the CLI, and would prefer something like PowerShell to have been the norm (with the accompanying toolset). Also note that there's nothing about the GUI that prevents someone from typing some commands too -- and in fact that's part of how lots of GUIs work.

Also, I wouldn't point to "modern web development toolchains" as something to advertise CLIs -- what would that be, Gulp, Grunt, Webpack and the like? All crufty solutions to inadequacies in steering the underlying language properly.

Even in theory, CLIs are not inherently more powerful, it's the other way around (they lack a dimension that GUIs offer (graphics) -- GUIs on the other hand don't lack any dimension -- they can incorporate text commands and text fields and pipelines just fine.

Re:Posix Has Become Outdated (2016) [pdf] by AnonymousPlanet on Saturday February 11, 2017 @11:13AM (#13623187)
Most Windows programmers I know have never really used an environment that doesn't punish you for using the command line. So what's your point? That most people you know never realised that many of their daily tasks can be easily automated if they wouldn't think that endless, tedious clickfests are the norm, and that this somehow proves that the CLI is obsolete?
Re:Posix Has Become Outdated (2016) [pdf] by coldtea on Saturday February 11, 2017 @06:24PM (#13625484)
>Most Windows programmers I know have never really used an environment that doesn't punish you for using the command line. So what's your point?

That they don't miss it either, and that the "endless, tedious clickfests" are a myth of people who don't know the other side.

Re:Posix Has Become Outdated (2016) [pdf] by AnonymousPlanet on Sunday February 12, 2017 @05:46AM (#13627784)
It's not a myth if you watch it happening nearly every day.
Re:Posix Has Become Outdated (2016) [pdf] by JdeBP on Saturday February 11, 2017 @09:38AM (#13622784)
I doubt that there is anyone at all in this entire discussion that can state with authority what "most Windows programmers" do. We can point to what we and the people we have met and talked to about this do, but to go from that comapratively scant few handsful of people to a statement the size of "most windows programmers" is an overgeneralization by several orders of magnitude.
Re:Posix Has Become Outdated (2016) [pdf] by coldtea on Saturday February 11, 2017 @06:30PM (#13625509)
I doubt that discussion and insight can happen without generalising. I also doubt that we need to cover 100% or even 99.999% of Windows programmers to be able to speak about what "most Windows programmers do".

Anybody who has experience from 2-5 different Windows shops and been to a few relevant conferences can make a quite accurate assessment of what "most Windows programmers" do.

Re:Posix Has Become Outdated (2016) [pdf] by umanwizard on Saturday February 11, 2017 @12:33PM (#13623613)
I write client-side code in Xcode but I use the terminal all day for various things.
Re:Posix Has Become Outdated (2016) [pdf] by dualogy on Saturday February 11, 2017 @06:29AM (#13622173)
Silly flamefest this. Even back when I worked mostly for Windows/MS shops or clients, writing CLI tooling for business (not developer) needs were often the order of the day. Batch import/export tools and what not. Also prototypes of daemon-like Windows Services for client testing. All kinds of tools that weren't directly developer tools. Sure enough, we were interfacing with IT folks at the client end but still, no programmers per se!

There's a reason GUI starts with a G. Other than "graphical" user interfaces, "other" user interfaces will also always have their place..

In some ways looking at how much non-developer computer usage in the past decades has been in front of excel/access sheets and tables, these aren't so "graphical" either. They essentially end up evolving into (haphazard) custom CLI-like keyboard-driven power tools, half-programmable, half requiring intimate familiarity with its "commands" (formulas) or even "training" to use them fluidly.

But if you'd rather code out your CLI needs in a PL REPL (aka CLI), more power to you ;D

Re:Posix Has Become Outdated (2016) [pdf] by lomnakkus on Saturday February 11, 2017 @07:31AM (#13622331)
Yes, it has.

I mean there may not be as many users, but they are massively more powerful (in the realm of data processing only, alas) than people who don't know the CLI. It may be obvious to us, but I should perhaps also point out that CLI ~= REPL for the language that you're using (Bash or whatever).

Sure, these days CLI may mean "a Jupyter notebook", but the same principle applies.

It may not always look the same, but the point is that a CLI/REPL should be a combinatorial force multiplier. Otherwise, it's not really worthwhile.

Re:Posix Has Become Outdated (2016) [pdf] by Gravityloss on Saturday February 11, 2017 @06:51AM (#13622222)
Interesting, I assume you do development? How do you handle scripting, moving files, grepping logs etc?
Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @07:07AM (#13622262)
GUI tooling? Languages like Python?
Re:Posix Has Become Outdated (2016) [pdf] by umanwizard on Saturday February 11, 2017 @12:43PM (#13623691)
How would you do this with GUI tooling:

Find all files containing a particular string in a directory tree. For each of those files, if its filename appears in a file called Whitelist, write it to a file called Output. Otherwise, do nothing.

Here is one possible command line version (I haven't tested it as I'm on mobile):

TMPF=`mktemp /tmp/XXXXXXXX` && ag MyString | awk -F: '{print $1}' | sort | uniq > $TMPF && comm -12 Whitelist $TMPF > Output && rm $TMPF

Re:Posix Has Become Outdated (2016) [pdf] by hawski on Saturday February 11, 2017 @01:21PM (#13623882)
You probably could use process substitution to get rid of the temp file:

  comm -12 Whitelist <(ag MyString | awk -F: '{print $1}' | sort | uniq) > Output
Then you could use unique flag of sort:

  comm -12 Whitelist <(ag MyString | awk -F: '{print $1}' | sort -u) > Output
Also ag can list only matching files (getting rid of awk and unique):

  comm -12 Whitelist <(ag -l MyString | sort) > Output
Re:Posix Has Become Outdated (2016) [pdf] by umanwizard on Saturday February 11, 2017 @01:48PM (#13624054)
Thanks for the tips!
Re:Posix Has Become Outdated (2016) [pdf] by pjmlp on Saturday February 11, 2017 @03:25PM (#13624600)
Apparently you forgot to read the part where I mentioned "Languages like Python?".

    import glob
    import shutil
     
    whitelist =  set(open('Whitelist').read().split())
    for fd in [ x for x in glob.glob('mydir/*', recursive=True) if x in whitelist]:
       shutil.copy(fd, 'OUTPUT')

Better, no need to create temporary files.
Re:Posix Has Become Outdated (2016) [pdf] by belltaco on Saturday February 11, 2017 @02:33PM (#13624289)
Would be trivial to write a GUI program to do the same.
Re:Posix Has Become Outdated (2016) [pdf] by umanwizard on Saturday February 11, 2017 @02:52PM (#13624395)
I think it would take at least a half hour, unless you have directory traversal, file I/O, and regex APIs memorized which most people don't. That's "trivial" in the grand scheme of things but it's really not practical to do day-in and day-out every time some random scripting necessity arises.

And you presumably have to clutter up your filesystem with random project files, etc, which is annoying. Are you going to have 1000 of these, one for every time you need to do some sort of minor task like this?

That command, on the other hand, took me 2 or 3 minutes to write, and I had to look up the syntax for `mktemp`. If I had remembered the mktemp syntax it would have taken a few seconds.