Re:RethinkDB internals: handling stack overflow on custom st...
by coffeemug on Friday December 31, 2010 @05:52AM
(#2055118)
I am not sure why they are writing their own library
We aren't, our code is based on a slightly modified version of libcoroutine. We've just added a few bells and whistles here and there to do a little bit more than what the library offers out of the box.
I don't think they are doing a ton of socket IO, so perhaps that is where their need to reinvent comes from.
Our real-world load can easily change between between disk bound, network bound, and CPU bound, so we need to support both IO bound and CPU bound workloads really well.
I agree that they are doing a bit of wheel-reinventing. I have used a couple existing coroutine libraries (libcoro from the Coro perl module, and libtask), and they both do a good job of conserving memory nicely. It's easy to write 10 lines of "threaded" Perl that can manage 30,000 slow HTTP connections with minimal overhead (a few k per fd).
I am not sure why they are writing their own library, but I do know that I have never used existing libraries for CPU-intensive purposes, only for IO-intensive purposes. I don't think they are doing a ton of socket IO, so perhaps that is where their need to reinvent comes from.