News for hackers. Stuff that matters.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by coldtea on Wednesday January 16, 2019 @08:41AM (#18920706)
Well, FreeBSD does it by calling a sh builtin:

  		/*
  		 * We are the child; make /bin/sh expand `words'.
  		 */
  		(void)__libc_sigprocmask(SIG_SETMASK, 
  &oldsigblock, NULL);
  		if ((pdes[1] != STDOUT_FILENO ?
  		    _dup2(pdes[1], STDOUT_FILENO) :
  		    _fcntl(pdes[1], F_SETFD, 0)) < 0)
  			_exit(1);
  		if (_fcntl(pdesw[0], F_SETFD, 0) < 0)
  			_exit(1);
  		execl(_PATH_BSHELL, "sh", flags & WRDE_UNDEF ? "-u" : "+u",
  		    "-c", "IFS=$1;eval \"$2\";"
  		    "freebsd_wordexp -f \"$3\" ${4:+\"$4\"}",
  		    "",
  		    ifs != NULL ? ifs : " \t\n",
  		    flags & WRDE_SHOWERR ? "" : "exec 2>/dev/null",
  		    wfdstr,
  		    flags & WRDE_NOCMD ? "-p" : "",
  		    (char *)NULL);
  		_exit(1);
    	}
Re:Libc on macOS invokes Perl as a subprocess for string pro... by self_awareness on Wednesday January 16, 2019 @09:46AM (#18921237)
Seems Linux version also depends on forking to the shell:

https://code.woboq.org/userspace/glibc/posix/wordexp.c.html

Re:Libc on macOS invokes Perl as a subprocess for string pro... by leni536 on Wednesday January 16, 2019 @10:08AM (#18921413)
For the sake of completeness musl libc also does this.

https://git.musl-libc.org/cgit/musl/tree/src/misc/wordexp.c#...

Re:Libc on macOS invokes Perl as a subprocess for string pro... by e12e on Wednesday January 16, 2019 @11:30AM (#18922215)
I guess the lesson is one might not want to implement word expansion in a shell using libc... [ed: if one has ambitions to function as /bin/sh...]
Re:Libc on macOS invokes Perl as a subprocess for string pro... by w0utert on Wednesday January 16, 2019 @11:25AM (#18922172)
But why? Looking at the wordexp synopsis it doesn't seem like a particularly advanced function. Why don't the various libc's just implement this function directly in C?
Re:Libc on macOS invokes Perl as a subprocess for string pro... by anyfoo on Wednesday January 16, 2019 @03:16PM (#18924641)
Because it's literally meant to perform shell style substitution, with all the myriad of expansions that the shell supports. Similarly to system(), which also passes its input to the shell, it's never safe to use with untrusted input. (The man-page says so.)
Re:Libc on macOS invokes Perl as a subprocess for string pro... by rurban on Thursday January 17, 2019 @04:28AM (#18929485)
The macOS libc is the FreeBSD libc. FreeBSD is upstream. FreeBSD changed it recently to use /bin/sh instead of /usr/bin/perl, because it's a bit smaller and faster.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by MichaelMoser123 on Wednesday January 16, 2019 @08:47AM (#18920762)
funny that glob doesn't do that http://xr.anadoxin.org/source/xref/macos-10.14-mojave/Libc-1... however wordexp does.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by coldtea on Wednesday January 16, 2019 @09:04AM (#18920897)
Glob is easier to code than wordexp. Much fewer rules.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by kazinator on Thursday January 17, 2019 @02:02AM (#18928931)
wordexp has to perform command substitution $(command args ...) as well as arithmetic expansion $((a + b * c)).
Re:Libc on macOS invokes Perl as a subprocess for string pro... by fouronnes3 on Wednesday January 16, 2019 @10:29AM (#18921615)
Slightly OT, you know how people hated on C++ ranges for readability? Honestly this messy soup of C is no better.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by giornogiovanna on Wednesday January 16, 2019 @11:02AM (#18921951)
What in particular upsets you about it? It's pretty readable to me, and unlike C++ ranges, is actually fairly terse.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by fouronnes3 on Wednesday January 16, 2019 @11:11AM (#18922034)
Cryptic variable names, weird indentation, ternary operator, string escaping, statements without braces.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by sjmulder on Wednesday January 16, 2019 @12:06PM (#18922546)
> weird indentation

This is regular BSD style, used in lots of places. Tabs for indentation, 4 spaces for wrapped lines.

> ternary operator

Totally fine here IMO. Would get really verbose without.

> without braces

Again, regular BSD style and a matter of taste.

Re:Libc on macOS invokes Perl as a subprocess for string pro... by taoistextremist on Wednesday January 16, 2019 @01:30PM (#18923474)
Yeah, but I think they have a point about the variable names. Really hard to tell the significance of anything there.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by kazinator on Thursday January 17, 2019 @01:33AM (#18928822)
Looks transparent to me. pdes must be "pipe descriptors". ifs probably refers to the IFS concept in the shell; it may have come from getenv("IFS"). wordexp is described as influenced by IFS, and this function may have to look at that variable even though it's calling an external program to pass it different arguments (which is a bit weird at first glance: the program seems dedicated to the purpose; why can't it handle that aspect internally). flags is probably coming from the wordexp flags argument; it's being tested using the documented bitmask constants in the API.

Basically the coder here didn't just make up identifiers; they have connections to pervasive Unix concepts.

Re:Libc on macOS invokes Perl as a subprocess for string pro... by kazinator on Thursday January 17, 2019 @01:29AM (#18928811)
How would you do this in C++ without string escaping, or extra memory allocation for string processing? Double quotes have to be passed embedded in the arguments passed to the shell.

One way (in C or C++) would be to define constants:

    #define DQ "\""
Then instead of "\"blah\" we have DQ "blah" DQ.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by kazinator on Thursday January 17, 2019 @01:17AM (#18928778)
It's valid C++, and you will see its ilk in C++ code bases.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by self_awareness on Wednesday January 16, 2019 @08:09AM (#18920488)
Seems it's very old code. New Libc doesn't do this. This is Libc from Mojave:

http://xr.anadoxin.org/source/xref/macos-10.14-mojave/Libc-1...

The 'perl' code was a part of Libc v825.24, which seems to be included between 10.7 (Lion) and 10.8 (Mountain Lion).

Of course I still find it hilarious that even the old code did that!

Re:Libc on macOS invokes Perl as a subprocess for string pro... by opencl on Wednesday January 16, 2019 @08:33AM (#18920648)
The current version replaces the perl subprocess with an sh subprocess. Doesn't seem like much of an improvement.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by anyfoo on Wednesday January 16, 2019 @03:15PM (#18924617)
Well, wordexp's purpose is literally to "perform shell-style word expansions", as quoted from the man page. It even supports command substitution if you don't pass WRDE_CMDSUB.

So really, the entire premise of that POSIX function is horrible[1]. Just like system(), which also explicitly executes the given command line using the shell. These functions are not safe to use with untrusted input (e.g. remotely), ever.

EDIT: [1] But arguably only as horrible as calling out to the shell is in general. If you e.g. use it as part of a shell utility that assumes full POSIX-permissioned access to your user anyway, it's not unreasonable because there isn't any privilege escalation at all. Though I'd argue that in the case of system() it's probably more clear to the developer that a shell callout is happening. And also, that the "shell-style" expansion performed here is kinda muddily defined.

Re:Libc on macOS invokes Perl as a subprocess for string pro... by AHTERIX5000 on Wednesday January 16, 2019 @05:11PM (#18925975)
OTOH the standard itself mentions implementing the function using shell in a subprocess [1]. POSIX is full of nastiness such as this, hopefully not too many maintained & used software actually needs wordexp.

[1] https://pubs.opengroup.org/onlinepubs/9699919799/functions/w...

Re:Libc on macOS invokes Perl as a subprocess for string pro... by panic on Wednesday January 16, 2019 @08:12AM (#18920508)
Discussed in 2015 (the code was out of date even then): https://news.ycombinator.com/item?id=9025572
Re:Libc on macOS invokes Perl as a subprocess for string pro... by josteink on Wednesday January 16, 2019 @07:32AM (#18920237)
That's obviously wrong.

It should use Emacs instead.

Re:Libc on macOS invokes Perl as a subprocess for string pro... by protomyth on Wednesday January 16, 2019 @08:22AM (#18920574)
I’m actually surprised there isn’t a library called libemacs. It would fulfill the mythos and be really useful for a lot of tools.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by gnulinux on Wednesday January 16, 2019 @09:01PM (#18927658)
Emacs doesn't do anything in C other than running lisp, emacs really is just a lisp implementation with GUI and text editing focus. libemacs would be just like lua or cpython.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by kazinator on Wednesday January 16, 2019 @11:06PM (#18928290)
Emacs has over a quarter million lines of C, which must be there for a reason.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by olliej on Wednesday January 16, 2019 @10:23AM (#18921566)
But then companies could use it to assault rms’s Freedom ;)
Re:Libc on macOS invokes Perl as a subprocess for string pro... by coldacid on Wednesday January 16, 2019 @02:23PM (#18924076)
All it would (need to) be is just a ELisp interpreter.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by greggyb on Thursday January 17, 2019 @03:00PM (#18934047)
Guile's VM already supports this.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by fouronnes3 on Wednesday January 16, 2019 @07:50AM (#18920338)
Perl doesn't depend on libc?
Re:Libc on macOS invokes Perl as a subprocess for string pro... by detaro on Wednesday January 16, 2019 @07:53AM (#18920351)
It does, but that's "not a problem" as long as it doesn't use this function to implement something that's executed during this function.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by jslabovitz on Wednesday January 16, 2019 @10:56AM (#18921879)
Turns out this is actually documented in the manpage for wordexp()! (And refers to the mentioned fact that it now calls 'sh' directly.)

> BUGS

> Do not pass untrusted user data to wordexp(), regardless of whether the WRDE_NOCMD flag is set. The wordexp() function attempts to detect input that would cause commands to be executed before passing it to the shell but it does not use the same parser so it may be fooled.

> The current wordexp() implementation does not recognize multibyte characters, since the shell (which it invokes to perform expansions) does not.

Re:Libc on macOS invokes Perl as a subprocess for string pro... by senozhatsky on Wednesday January 16, 2019 @07:33AM (#18920244)
Shall somebody send a pull request [0]

    -    /* XXX this is _not_ designed to be fast */
    +    /* XXX this is _not_ designed to be safe */
[0] https://github.com/Apple-FOSS-Mirror/Libc/blob/2ca2ae7464771...
Re:Libc on macOS invokes Perl as a subprocess for string pro... by api on Wednesday January 16, 2019 @08:32AM (#18920640)
/* This function computes the expansion rate of spacetime. This version contains an additional factor that causes it to accelerate to allow for rapid testing of other aspects of the physics engine. This MUST be removed before the production release or spacetime will accelerate forever and experience heat death instead of reaching steady state. -God */
Re:Libc on macOS invokes Perl as a subprocess for string pro... by acura on Wednesday January 16, 2019 @08:02AM (#18920419)
Isn't the line bellow that line enough? /* wordexp is also rife with security "challenges",
Re:Libc on macOS invokes Perl as a subprocess for string pro... by senozhatsky on Wednesday January 16, 2019 @08:33AM (#18920647)
My assumption was that those security "challenges" were related to expansion (wordexp()) per se; not to the way wordexp() was implemented in this particular case.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by MichaelMoser123 on Wednesday January 16, 2019 @08:07AM (#18920467)
Now I see why you can't change /usr/bin on macos. actually there is both perl5 and python2.7 in /usr/bin, libc does have a choice (that is if the tweet is true)...

https://en.wikipedia.org/wiki/System_Integrity_Protection

Re:Libc on macOS invokes Perl as a subprocess for string pro... by olliej on Wednesday January 16, 2019 @10:28AM (#18921609)
You can’t change use/bin because that’s a common malware attack vector.

It also has the nice effect of forcing user installed utilities to install in the /local/ variants (which user build projects should be doing on Linux iirc), so an OS update doesn’t overwrite user data.

Re:Libc on macOS invokes Perl as a subprocess for string pro... by tinus_hn on Wednesday January 16, 2019 @05:00PM (#18925879)
Also if it bothers you, you can just turn it off. It's just difficult enough people can't simply put some screenshots on a webpage to work around it because they are too lazy to code properly.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by fit2rule on Wednesday January 16, 2019 @08:06AM (#18920458)
I find it hard to believe that there's any software out there that doesn't, eventually, invoke Perl as a subprocess .. I mean, its Perl.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by leejo on Wednesday January 16, 2019 @08:20AM (#18920563)
That's essentially what the OG tweet is saying: Pinnacle of software development: you can solve the problem with three lines of Perl, but you don’t, because of a non-argument against Perl. Since there didn't used to be that many arguments against perl/Perl it worked its way into a lot of systems even if it wasn't actually implementing the system.

Of course Perl, having fallen out of vogue, probably wouldn't be used today but it used to be everywhere so its footprint is still pretty large.

Also - I can't help but see the irony in shelling out to perl given experienced Perl developers always tell the less experienced ones to avoid shelling out from Perl if possible and to only do that as a last resort if there isn't an existing library to solve the problem.

Re:Libc on macOS invokes Perl as a subprocess for string pro... by virgilp on Wednesday January 16, 2019 @10:23AM (#18921560)
> experienced Perl developers always tell the less experienced ones to avoid shelling out from Perl if possible

The reason for that is obvious, right? By induction, shelling out from Perl would only result in the called process shelling back to Perl. So it's much better to just call that Perl code directly.

Re:Libc on macOS invokes Perl as a subprocess for string pro... by rurban on Thursday January 17, 2019 @04:38AM (#18929521)
Only loud-mouthed inexperienced middle managers tell their juniors to follow NIH and do everything in pure perl. It's pure fear to be broken by changed dependencies.

More experienced managers tell their devs to shell out to standard tools like sh, wget, curl, dig, mysqlclient and not use the builtin pure-perl libs. The tools are much better, the code is 10x smaller and faster, and you are getting updates for free (e.g. ssl). Even in C I very often call system("wget http://...") and avoid libcurl.

Re:Libc on macOS invokes Perl as a subprocess for string pro... by leejo on Thursday January 17, 2019 @04:48AM (#18929557)
Only loud-mouthed inexperienced middle managers tell their juniors to follow NIH and do everything in pure perl. It's pure fear to be broken by changed dependencies.

It's not NIH if you're advocating using CPAN modules or existing libraries.

Sure, use the right tool - but don't advocate to juniors a method that can lead to OS command injection because they're not experienced enough to know to, or how to, sanitise their inputs.

Re:Libc on macOS invokes Perl as a subprocess for string pro... by the_mitsuhiko on Wednesday January 16, 2019 @10:46AM (#18921790)
Not exactly related to the link but apparently the author of that tweet blocked me. I don’t recall having ever had any interactions with them. Is there a way to contact that person and figure out why? I have no idea what I did and I’m quite puzzled.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by woodman on Wednesday January 16, 2019 @12:59PM (#18923139)
There is a good chance they subscribe to a blocklist, so you could be blocked by anyone of a thousand people. Image the old PGP web of trust, but for crafting perfect echo chambers. I wonder if anybody has ever done the math on that.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by stonogo on Wednesday January 16, 2019 @10:57AM (#18921896)
You find out someone has blocked you, and your instinct is to communicate with them? The specific thing they have explicitly disallowed? I'd reconsider this, and just move on.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by the_mitsuhiko on Wednesday January 16, 2019 @11:05AM (#18921970)
Is that how you are supposed to deal with that? How is one supposed to improve as a human if one does not know where their mistakes are? If I did something wrong I would like to know.

I did not contact that person anyways and I doubt there is a way. But at least knowing when I was blocked could help me deduce what might be the reason.

Honestly I do find this quite upsetting.

Re:Libc on macOS invokes Perl as a subprocess for string pro... by Shish2k on Wednesday January 16, 2019 @01:42PM (#18923621)
> Is that how you are supposed to deal with that?

I believe the commonly accepted answer is "Yes, it's not my job to give you the necessary information to improve yourself". The same people who say that also often ask "Why aren't the people around me improving themselves?" :P

(Personally, I would prefer it if people pointed out my mistakes, and I do the same for others as a courtesy, not an obligation. I do understand if they don't have the energy to do that, but I think if they don't, then they forfeit the right to complain about a lack of improvement)

Re:Libc on macOS invokes Perl as a subprocess for string pro... by the_mitsuhiko on Wednesday January 16, 2019 @03:16PM (#18924640)
> I believe the commonly accepted answer is "Yes, it's not my job to give you the necessary information to improve yourself". The same people who say that also often ask "Why aren't the people around me improving themselves?"

But that’s in no way comparable. When in the real world i screw up I can tell from people’s responses. Body language, their actions etc. In this case it’s just discovering at one point someone blocked one without any indication of when that happened and why without any indication.

//edit: also even weirder in an effort to see where our interactions might have been i found a tweet from 2014 by myself about the same topic: https://mobile.twitter.com/mitsuhiko/status/5264923088676700...

Re:Libc on macOS invokes Perl as a subprocess for string pro... by philwelch on Wednesday January 16, 2019 @12:00PM (#18922483)
It could just be irrational intolerance on the part of the other person.
Re:Libc on macOS invokes Perl as a subprocess for string pro... by acura on Wednesday January 16, 2019 @08:04AM (#18920435)
Not good looking at all but, latest commit to that repo is Updated on Oct 11, 2012.

So how well does it reflect reality?

Re:Libc on macOS invokes Perl as a subprocess for string pro... by raimue on Wednesday January 16, 2019 @09:09AM (#18920935)
The official sources are published by Apple on https://opensource.apple.com

This repository is just a snapshot that somebody else prepared and uploaded to GitHub, but apparently it is not maintained.