News for hackers. Stuff that matters.
Re:Parallelizing the Naughty Dog Engine Using Fibers [video] by corysama on Thursday May 26, 2016 @06:25PM (#11782184)
There's no official support for fibers in Posix. There are lots of implementations anyway, but it's hard to get completely right. Long ago, the Win32 team got tired of people rolling their own subtley incorrect/incomplete fiber implementations for Windows and added official support to put an end to that support issue https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...
Re:Parallelizing the Naughty Dog Engine Using Fibers [video] by btrask on Thursday May 26, 2016 @07:47PM (#11782690)
Actually, there is: https://en.wikipedia.org/wiki/Setcontext

A library I personally recommend, which supports several platforms and back-ends: http://byuu.org/library/libco/

Re:Parallelizing the Naughty Dog Engine Using Fibers [video] by david-given on Friday May 27, 2016 @04:31AM (#11784662)
A few years back I built an app which used coroutines extensively (<plug> greylisting SMTP proxy, now abandoned because I don't run my own mailserver any more, http://cowlark.com/spey/ </plug>). I originally used getcontext/setcontext.

Superficially it was really easy, but turned into a goddamned nightmare. The problem is that pthreads gets on really badly with user-allocated stacks; some pthreads implementations find the current thread ID by looking at a magic value on the current thread's stack, and of course with a user-allocated stack this wasn't there.

But why was I using pthreads alongside coroutines, you ask? Well, I wasn't. But I was linking to a library which was referring to pthreads symbols, even though it wasn't calling them; which made the linker automatically pull in the thread-safe glibc, so malloc/free ended up wrapping in mutexes...

The net effect was bizarre and infrequent crashes that only showed up on machines which weren't mine and that I didn't have access to. I spent months debugging that.

Eventually I switched to simulating coroutines using pthreads with a big lock so that only one thread could run at a time. (Which actually turned out to be better, because now I could do some operations properly in parallel.)

Beware of setcontext! Bewaaaaaare!

Re:Parallelizing the Naughty Dog Engine Using Fibers [video] by sillysaurus3 on Thursday May 26, 2016 @08:10PM (#11782847)
setcontext isn't Posix, so corysama is the best kind of correct. But setcontext exists on most flavors of Linux, FreeBSD, and OS X, so you can basically assume it's supported.
Re:Parallelizing the Naughty Dog Engine Using Fibers [video] by gpderetta on Friday May 27, 2016 @07:09AM (#11785186)
setcontext was posix until POSIX.2008.
Re:Parallelizing the Naughty Dog Engine Using Fibers [video] by byuu on Friday May 27, 2016 @11:56PM (#11790516)
Just a fair warning, setcontext is horrendously slow. It seems to be doing kernel transitions or something. It's about 600 times slower than longjmp with jmpbuf stack pointer hacking, which is itself already about twice as slow as an ideal assembly version.
Re:Parallelizing the Naughty Dog Engine Using Fibers [video] by shinymark on Thursday May 26, 2016 @09:32PM (#11783257)
Our team just got hit by this. We are using fibers in a custom C++ game engine without issue on Windows, OSX, and some console platforms. Recently we began the work to bring the code base to iOS. Guess what? No fiber support. Doh!
Re:Parallelizing the Naughty Dog Engine Using Fibers [video] by corysama on Thursday May 26, 2016 @10:11PM (#11783420)
Looks [1] like http://byuu.org/library/libco/ (mentioned below) might work on iOS. I would definitely be interested in fiber libs that are confirmed to work on iOS and/or Android if anyone knows any. libco looks like it might fit the bill. But, it's not completely clear and I have not tried it.

[1] http://libretro.com/forums/archive/index.php?t-785.html

Re:Parallelizing the Naughty Dog Engine Using Fibers [video] by clevernickname on Thursday May 26, 2016 @10:38PM (#11783503)
libco relies on being able to map WX pages for a rather silly reason[1], so it won't run on iOS without some work.

[1] The author didn't like requiring assemblers to build his projects, so instead the coroutine switch machine code is embedded in the binary as a byte array, which is then copied to a page with WX permissions when the library is initialized.

https://github.com/creationix/libco/blob/master/amd64.c

Re:Parallelizing the Naughty Dog Engine Using Fibers [video] by byuu on Saturday May 28, 2016 @12:02AM (#11790536)
Why do you feel [1] is a silly reason? You can build my software right now in one step: "install TDM-GCC"; anything additional complicates things and reduces portability. There's also versions that don't require assembler code at all (setjmp/longjmp version), and you can also make an ios.c version that has inline assembler since that target will only ever have a single compiler anyway.
Re:Parallelizing the Naughty Dog Engine Using Fibers [video] by clevernickname on Saturday May 28, 2016 @04:13PM (#11793258)
Ironic in light of your comments on the OpenBSD forced W^X thing, but I happen to agree that strict W^X should be used everywhere possible. It may not matter for higan (though it would be funny to see someone exploit it), but it could be a deal breaker for more security-conscious projects that might want to use libco (say, for a cothreaded server).

As for assemblers, I just don't see how requiring gas or maybe NASM/YASM for x86 would complicate the build process so much. Heck, gas is a dependency of gcc. The only problem I see is for Visual Studio users, which could be solved with using NASM/YASM for Windows (which is not really a big deal when the co_swap_function has to be different due to the calling conventions anyway), or a separate MASM implementation for Visual Studio (giving you duplicate implementations with VS Windows and gcc Windows on x86 and x86_64 respectively).

I know you're consistent with your views on code clarity and that this is a hobby for you etc, but libco seems like a tiny, static enough library that it would make sense to accept a little ugliness in order to Do The Right Thing™.

(PS: Thank you for higan, I've learned so much from it over the years.)

Re:Parallelizing the Naughty Dog Engine Using Fibers [video] by byuu on Sunday May 29, 2016 @10:23PM (#11799213)
> I happen to agree that strict W^X should be used everywhere possible

I agree. I just feel there are exceptions, one of which is dynamic recompiling emulators.

> it could be a deal breaker for more security-conscious projects that might want to use libco (say, for a cothreaded server).

I was worried that since mprotect works on entire pages, that by clearing the write flag, I might be removing writability to certain variables. But, theoretically, static char arrays should always end up in the .data section, so it should be fine. Still feel uneasy about it.

Unfortunately, there's no POSIX way to read the current protections, so I can't just apply |X; plus that's probably not safe.

I could allocate three _SC_PAGESIZE blocks of memory (for alignment), mark the middle block R|W, copy the function to it, and then mark the block R|X. As long as OpenBSD allows those two calls, that should work.

> it would make sense to accept a little ugliness in order to Do The Right Thing™

I don't see how requiring Windows users to download yasm (64-bit) or nasm (32-bit) is doing the right thing. Every single additional step required to compile my software will result in more and more people giving up or not bothering.

Re:Parallelizing the Naughty Dog Engine Using Fibers [video] by shinymark on Friday May 27, 2016 @12:48AM (#11783963)
We ended up integrating some Boost ARM assembly code that did the trick (both 32 and 64 bit). I don't personally have the details because I didn't do the integration but after a few iterations and some tweaks it worked for us.
Re:Parallelizing the Naughty Dog Engine Using Fibers [video] by byuu on Friday May 27, 2016 @11:59PM (#11790524)
It has an ARM implementation, but there are lots of subtle ARM variations. You'd probably have to tweak it a tad, but the core ARM cothread switching code is a mere three instructions, so it shouldn't be too hard to adapt:

    0xe8a16ff0,  /* stmia r1!, {r4-r11,sp,lr} */
    0xe8b0aff0,  /* ldmia r0!, {r4-r11,sp,pc} */
    0xe12fff1e,  /* bx lr                     */
If you do, send me a patch and I'll merge it upstream.