News for hackers. Stuff that matters.
Re:C library system-call wrappers, or the lack thereof by int_19h on Tuesday November 13, 2018 @11:31PM (#18447494)
This is one area where I feel that BSD approach (where the same team maintains both the kernel and the libc, and they're shipped in sync as part of the same release) makes a lot more sense.

In fact, come to think of it, Linux is the only OS where syscalls are the official public userspace API, is it not? On all other platforms, they're an implementation detail behind the system libraries.

Re:C library system-call wrappers, or the lack thereof by sanxiyn on Wednesday November 14, 2018 @12:22AM (#18447707)
syscall-as-API does have some advantages in that you can avoid libc if you want to. One example is Go's Linux target; on Linux, Go standard library calls syscalls directly without any help of libc.
Re:C library system-call wrappers, or the lack thereof by kayamon on Wednesday November 14, 2018 @12:46AM (#18447788)
You can achieve that on other OSes too though, still without requiring libc.

For instance, on Windows the syscall API is hidden and changes from release to release, but KERNEL32.DLL persists as the stable system API that you're supposed to call.

Linux desperately needs to get past the notion that libc is the only gateway to the kernel, and instead start supplying a standardized "system/kernel" user-level wrapper library. One which wraps the syscalls nicely but doesn't try to add additional language-specific functionality like printf or memcpy.

Re:C library system-call wrappers, or the lack thereof by vetinari on Wednesday November 14, 2018 @04:46AM (#18448684)
> Linux desperately needs to get past the notion that libc is the only gateway to the kernel,

Are you sure about that? Linux doesn't have that notion, it keeps the syscal ABI stable, and anyone can use it directly, like Go does. There's no need to go through blessed syscall library, which on their respective systems are the only gateway to the kernel.

Re:C library system-call wrappers, or the lack thereof by masklinn on Wednesday November 14, 2018 @08:03AM (#18449513)
The biggest advantage is that it lets other OS easily impersonate linux e.g. SmartOS or Windows can provide kernel-level "linux persona" and have an unmodified linux userland run on top.
Re:C library system-call wrappers, or the lack thereof by int0x80 on Wednesday November 14, 2018 @10:15AM (#18450405)
They can do right now. Both the API and ABI are very rarely (read never but very problematic bugs or similar) broken in a backwards way.
Re:C library system-call wrappers, or the lack thereof by masklinn on Wednesday November 14, 2018 @11:26AM (#18451058)
That's my point. It's only possible with Linux because Linux provides a stable kernel ABI. You can't impersonate Windows or OSX the same way.
Re:C library system-call wrappers, or the lack thereof by int_19h on Wednesday November 14, 2018 @06:35PM (#18455322)
But you can impersonate Windows by providing your own implementation of the Kernel32 API. What's the essential difference between that, and emulating syscall API?
Re:C library system-call wrappers, or the lack thereof by int0x80 on Wednesday November 14, 2018 @11:30AM (#18451096)
Ahh sorry, I understood the other way around :)
Re:C library system-call wrappers, or the lack thereof by int_19h on Wednesday November 14, 2018 @02:31AM (#18448205)
On Windows, kernel32.dll is basically the Windows equivalent of libc - it's still a userspace library that wraps syscalls, and you're not supposed to use them directly.

But I don't see why it's a problem, either. What's the actual benefit of Go invoking syscalls directly on Linux? That it doesn't depend on glibc? But that is only an advantage because glibc is not guaranteed to be there on Linux, the way e.g. kernel32 is on Windows. If it were, there'd be no reason to not use it.

Re:C library system-call wrappers, or the lack thereof by vetinari on Wednesday November 14, 2018 @04:44AM (#18448673)
Specifically Go has very particular ideas about memory and stack layout, which do not go well with C libraries. In order to call a C library, Go has to go through thunking shims.

When using syscalls directly, it does not matter. Kernel doesn't care how your laid out your userspace stack.

Re:C library system-call wrappers, or the lack thereof by cesarb on Wednesday November 14, 2018 @06:03AM (#18449012)
> Kernel doesn't care how your laid out your userspace stack.

Except when it does: https://github.com/golang/go/issues/20427

Re:C library system-call wrappers, or the lack thereof by zaarn on Wednesday November 14, 2018 @07:38AM (#18449388)
IIRC this was because of a rather amusing behaviour of a Gentoo toolchain with some hardening patches, in this case doing a bit of fun on the stack to make sure it's safe.

A lot of the hardening patches come with disclaimers that some software might break, and as shown by this bug, for good reason.

I'd rather blame the compiler patches for this silly behaviour (and maybe the kernel for not documenting and limiting how much stack VDSO can use)

Re:C library system-call wrappers, or the lack thereof by ben0x539 on Monday November 26, 2018 @05:38PM (#18537503)
It's not really obvious to me that the original issue, mentioning os.Exec calls and Ubuntu versions, has the same cause as the crashes from the commenter mentioning patched Gentoo toolchains.
Re:C library system-call wrappers, or the lack thereof by bregma on Wednesday November 14, 2018 @10:55AM (#18450800)
"libc" is the C-language runtime library. The Go language is not the C language. Different things are different. There is no reason for programs written in the Go language to use the C-language runtime in any operating environment, any more than there is a requirement for all programming languages to use curly braces to delimit lexical scope.
Re:C library system-call wrappers, or the lack thereof by IshKebab on Wednesday November 14, 2018 @04:30AM (#18448615)
> That it doesn't depend on glibc? But that is only an advantage because glibc is not guaranteed to be there on Linux, the way e.g. kernel32 is on Windows. If it were, there'd be no reason to not use it.

Yes - but it's an advantage because glibc is a compatibility nightmare. One of Go's really nice features is that you can compile a binary and it will run pretty much anywhere. That would be pretty much impossible if it linked with glibc (even if you ignore the fact that glibc might not be present).

Re:C library system-call wrappers, or the lack thereof by 4ad on Wednesday November 14, 2018 @05:17AM (#18448818)
Not true. On Solaris (and Windows), and more recently in macOS as well we link with libc (or equivalent), and we can still cross compile from any system to these systems. It's not a free lunch of course, doing raw system calls would be preferable, for many reasons.

I implemented support for this, for ELF, when I wrote the Solaris port. Other people have done it for PE and Mach-O. It is a fallacy to think that you need access to a shared library in order to link with it. That's only true for C toolchains that don't know better. For Go we have our own toolchain and it doesn't have such a restriction.

This is one of the points lost to the "why didn't you just use LLVM?" crowds. Our own toolchain allows us flexibility that simply doesn't exist with traditional toolchains.

Re:C library system-call wrappers, or the lack thereof by matthewbauer on Wednesday November 14, 2018 @11:59AM (#18451373)
How do you handle changes to the kernel? Can your binaries run on any macOS version or only the very latest? This sounds really cool but also super dangerous. Apple is not known for providing a stable ABI :)
Re:C library system-call wrappers, or the lack thereof by 4ad on Wednesday November 14, 2018 @12:47PM (#18451876)
You got it backwards, Go used to do raw system calls on macOS, and binaries were occasionally broken by kernel updates. Now Go uses libc on macOS, and binaries are forward compatible with future macOS versions just like any other C/C++/ObjC/swift program. OS X 10.10 (Yosemite) is the current minimum supported version.
Re:C library system-call wrappers, or the lack thereof by int_19h on Wednesday November 14, 2018 @06:36PM (#18455331)
Unfortunately, Go still insists on using syscalls on the BSDs, despite them not being a stable (between major releases) API there either.

I hope they're planning to fix that...

Re:C library system-call wrappers, or the lack thereof by matthewbauer on Wednesday November 14, 2018 @02:35PM (#18453084)
Ok nice! In many ways it would be cool if apple had a stable interface for XNU like we do in Linux.
Re:C library system-call wrappers, or the lack thereof by swiley on Wednesday November 14, 2018 @06:30AM (#18449108)
This used to be true, Unfortunately it’s not now.

I found this out when trying to copy a go program I had compiled on alpine to an Ubuntu machine and got the “file not found” error from the linker. :(

Try it yourself: run ldd on a recent go binary.

Re:C library system-call wrappers, or the lack thereof by zaarn on Wednesday November 14, 2018 @07:41AM (#18449396)
Go uses the glibc for DNS resolution by default, if you set CGO_ENABLED=0 during compiling it should result in a static binary.
Re:C library system-call wrappers, or the lack thereof by btbuilder on Wednesday November 14, 2018 @03:19AM (#18448379)
As the article eludes to it is not without its challenges, however. As an example see this issue: https://github.com/golang/go/issues/1435 which goes into some of the details why implementing a call for setuid() is not straightforward without the knowledge built into glibc.

Also the trickiness of having efficient process fork/exec based on vfork: http://ewontfix.com/7/ and the considerations going into Go: https://go-review.googlesource.com/c/go/+/46173/

Re:C library system-call wrappers, or the lack thereof by pm215 on Wednesday November 14, 2018 @07:15AM (#18449278)
I think for setuid in particular the right fix is for the kernel to provide a new syscall that operates on the whole process. The current hoops that glibc has to jump through with signals are complicated (and maybe racy? it's been a while since I looked at the code) and tie up a signal for libc's private use; better support at the kernel layer would allow that to all be eventually dropped.
Re:C library system-call wrappers, or the lack thereof by carapace on Wednesday November 14, 2018 @12:00PM (#18451381)
( allude (elude means escape))
Re:C library system-call wrappers, or the lack thereof by masklinn on Wednesday November 14, 2018 @03:55AM (#18448502)
> In fact, come to think of it, Linux is the only OS where syscalls are the official public userspace API, is it not?

Although I never checked so I could be completely wrong, I would expect folks shipping just kernels like SeL4 to ship stable kernel ABI rather than a libc.

Re:C library system-call wrappers, or the lack thereof by Annatar on Wednesday November 14, 2018 @02:37AM (#18448229)
On all other platforms, they're an implementation detail behind the system libraries.

Yes and this is by design, out of necessity, taught by experience. Commercial customers back in the day paid lots and lots of money, so solutions had to be found and they had to work.

Re:C library system-call wrappers, or the lack thereof by saagarjha on Wednesday November 14, 2018 @01:46AM (#18448041)
macOS lists them all in /usr/include/sys/syscall.h, so I guess you can consider that public API?
Re:C library system-call wrappers, or the lack thereof by eridius on Wednesday November 14, 2018 @04:03AM (#18448528)
macOS syscalls are explicitly not stable and may change from release to release. The only supported interface to them on macOS is libc.
Re:C library system-call wrappers, or the lack thereof by akvadrako on Wednesday November 14, 2018 @02:00AM (#18448096)
That file doesn't even exist on my system, but in any case public means official, not in some header.
Re:C library system-call wrappers, or the lack thereof by saagarjha on Wednesday November 14, 2018 @02:11AM (#18448128)
If you have the command line tools installed, it should also be under /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/syscall.h. macOS, by default, does not ship with these headers at all, and as of macOS Mojave they no longer provide them under /usr/include unless you install a certain package.

With that being said, I disagree with your characterization of “some header”–anything that’s in Apple’s headers is public API, whether it has a fancy page on developer.apple.com or not. Apple has a very clear definition of what they consider to be “private”, and anything in /usr/include isn’t it.

Re:C library system-call wrappers, or the lack thereof by akvadrako on Wednesday November 14, 2018 @06:14AM (#18449052)
Actually the path is:

  /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sys/syscall.h 
and you can clearly see that the file is wrapped with:

  #ifdef __APPLE_API_PRIVATE
  ...
  #endif /* __APPLE_API_PRIVATE */
Re:C library system-call wrappers, or the lack thereof by saagarjha on Wednesday November 14, 2018 @11:56PM (#18456951)
Again, if you have the Command Line Tools installed, it'll be under /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/. Xcode also has a copy, but I think fewer people have that installed.
Re:C library system-call wrappers, or the lack thereof by amaccuish on Wednesday November 14, 2018 @08:46AM (#18449777)
Nope, on my system, without xcode, the previous poster's path is correct: /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/syscall.h
Re:C library system-call wrappers, or the lack thereof by int_19h on Wednesday November 14, 2018 @02:24AM (#18448170)
They need to be public and stable in order to be a usable userspace API for anything other than libc. I don't think the latter requirement is satisfied:

https://github.com/golang/go/commit/02118ad0a0cacb00c1d834b3...

Re:C library system-call wrappers, or the lack thereof by masklinn on Wednesday November 14, 2018 @08:05AM (#18449524)
Nope, in fact it's so absolutely not stable (or supported) that libSystem can't be statically linked to.