News for hackers. Stuff that matters.
Re:In C, how do you know if the dynamic allocation succeeded? by magicalhippo on Friday November 05, 2021 @08:09PM (#29126071)
> But what happens when the kernel needs to copy one and has run out of memory?

Don't allow it. Fork the process with read only pages except for the ranges passed to fork(). Count read-write pages as used memory by the child process.

If the forked process wants to write to a page that's read-only it'll have to do a system call to turn it read-write. That call can then fail if there's not enough free memory to copy the pages.

Of course, kinda hard to fix this now...

Re:In C, how do you know if the dynamic allocation succeeded? by tsimionescu on Saturday November 06, 2021 @01:06AM (#29127676)
> Don't allow it. Fork the process with read only pages except for the ranges passed to fork().

This doesn't work because of the horrible fork/exec design. If I am a huge process and I want to run `ls`, I will first have to clone myself using fork(), which may trigger an OOM, even if the first action of my clone would have been exec(), ignoring all of that memory.

I am always surprised that no one has added a sane 'spawn process' primitive to replace fork/exec. Especially since fork() without exec() only really works in single-threaded processes.

Re:In C, how do you know if the dynamic allocation succeeded? by naniwaduni on Saturday November 06, 2021 @01:22AM (#29127739)
We do have posix_spawn &c., which are expressly designed to be implementable even in situations where fork doesn't make sense.
Re:In C, how do you know if the dynamic allocation succeeded? by tsimionescu on Saturday November 06, 2021 @01:50AM (#29127857)
Interesting, I'd never learned about that. If I understand correctly, this actually used to be implemented as a fork()+exec() on many systems, though. It's actually vfork() that addressed the problem and allowed an actually useful implementation of posix_spawn in Linux at least.
Re:In C, how do you know if the dynamic allocation succeeded? by toast0 on Friday November 05, 2021 @08:49PM (#29126370)
You'd presumably need the parent to also do a system call to mark pages read-write after forking, and that would need to include the stack pages, and that sounds like lots of fun; especially since sometimes library functions fork (although most of that is to fork/exec and more specific apis for that exist now). Not that it isn't loads of fun anyway, but woe betide thee who forks and threads and makes the other threads have a read-only stack.

Probably you'd get people just marking as read-write and returning in sigsegv handlers, which I'm sure has great security properties... OTOH, at least there's an opportunity to deny the remap in the handler and get a decent crashdump from the program or a sliver of hope for managing the situation.

Re:In C, how do you know if the dynamic allocation succeeded? by pjc50 on Saturday November 06, 2021 @10:06AM (#29130302)
> Of course, kinda hard to fix this now..

Well, yes, if you can break backwards compatibility you can do anything. Except run all the existing software.

Re:In C, how do you know if the dynamic allocation succeeded? by enedil on Friday November 05, 2021 @08:35PM (#29126282)
That also means that if you used seccomp to restrict mprotect syscall, the child process may never ever write again, which is an unfortunate design choice.
Re:In C, how do you know if the dynamic allocation succeeded? by magicalhippo on Friday November 05, 2021 @09:19PM (#29126601)
Seccomp can check for the PROT_EXEC flag though, and let the others pass, no?