News for hackers. Stuff that matters.
Re:RethinkDB internals: handling stack overflow on custom st... by leif on Friday December 31, 2010 @05:05AM (#2055034)
> The point wasn't that you couldn't change this value, it's that for our purposes (lots of green threads, shallow stacks) a much smaller default stack size is more sensible.

Slava, you're better than that. WHY is a smaller default stack size more sensible? If this is all you need, and you can change it manually, do that!

Re:RethinkDB internals: handling stack overflow on custom st... by coffeemug on Friday December 31, 2010 @05:13AM (#2055055)
WHY is a smaller default stack size more sensible?

Because if you have a large number of threads on an architecture with a small address space, you take away memory space from the buffer cache which can have catastrophic effects on performance.

If this is all you need, and you can change it manually, do that!

This needs to be understood in the context of the previous posts. The issue with NPTL is significantly lower performance on large number of threads due to context switching. Dan implemented a solution that uses coroutines to address that problem. This post deals with the over-provisioning issue described above, and it starts out by describing the default configuration for pthreads. Adjusting this to a smaller number on the NPTL implementation isn't enough, we needed to adjust it for the coroutines implementation.

Re:RethinkDB internals: handling stack overflow on custom st... by cperciva on Friday December 31, 2010 @05:17AM (#2055058)
Because if you have a large number of threads on an architecture with a small address space, you take away memory space from the buffer cache which can have catastrophic effects on performance.

Say again?

Address space exhaustion is a problem, sure, but it has nothing to do with the buffer cache. The kernel's address space is completely separate from any particular process' address space.

Re:RethinkDB internals: handling stack overflow on custom st... by coffeemug on Friday December 31, 2010 @05:48AM (#2055112)
Address space exhaustion is a problem, sure, but it has nothing to do with the buffer cache. The kernel's address space is completely separate from any particular process' address space.

For various reasons we don't use the kernel's cache, we have our own cache implementation that's in the same address space as the rest of the process.

Re:RethinkDB internals: handling stack overflow on custom st... by cperciva on Friday December 31, 2010 @05:52AM (#2055117)
Oh dear god.

You have my sincere and profound sympathies.

Re:RethinkDB internals: handling stack overflow on custom st... by barrkel on Friday December 31, 2010 @05:55AM (#2055122)
It's pretty unusual for databases to delegate something as vital to performance as caching to the OS.
Re:RethinkDB internals: handling stack overflow on custom st... by cperciva on Friday December 31, 2010 @05:59AM (#2055128)
Sure, a database needs to cache. But it doesn't automatically need to cache at that level -- usually they can get away with caching data rather than pages.
Re:RethinkDB internals: handling stack overflow on custom st... by strlen on Wednesday January 05, 2011 @06:00PM (#2072946)
If you have your own cache and you don't use direct I/O i.e., they go through the OS page cache, don't have you have the issue of the OS page cache and the database "fighting" each other?