News for hackers. Stuff that matters.
Re:What's wrong with 1975 programming by barrkel on Wednesday July 28, 2010 @04:23PM (#1555538)
The major assumption is that the OS VMM's caching behaviour is as smart or smarter than the application's caching behaviour; that is, the operating system knows better than the application when to evict something from the cache. That may be true for many stupid applications, but I seriously doubt it's always the case.
Re:What's wrong with 1975 programming by shabble on Wednesday July 28, 2010 @04:27PM (#1555547)
I was informed recently in a Masters level course on database implementation that databases typically wish to explicitly circumvent a lot of this caching, especially for IO buffering, since they can do the same thing at the app level, but leverage the fact they have much more well-defined access patterns than some generic kernel system that has to perform adequately across all workloads.

I believe the O`_DIRECT flag can be used for this purpose, although a quick bit of googling suggests it's getting flack from all over the place, including Linus himself.

Re:What's wrong with 1975 programming by btilly on Wednesday July 28, 2010 @07:55PM (#1556065)
Another big reasons that databases with to circumvent this caching is that they really, really need to make sure that when a transaction is committed, that they know what is on disk. Having the OS pretend stuff is written to disk is great for performance, but not so great when you're trying to recover after someone pulled the power.
Re:What's wrong with 1975 programming by dedward on Wednesday July 28, 2010 @09:25PM (#1556244)
This is where you get DBs that use their own raw block devices, as well as keeping transaction logs on other regular filesystems for integrity.... Oracle does something similar no?
Re:What's wrong with 1975 programming by dfox on Thursday July 29, 2010 @05:49AM (#1557083)
For example PostgreSQL actually has almost none caching built-in and almost exclusively relies on OS (in default and recommended configuration).

I expect that most of DBMS that wish to circumvent OS-level caches do this because (a) it was good idea when they was first implemented and (b) use some highly-peculiar on-disk layout that might actually benefit from this (modern databases tend to not have any special "well-defined" access patterns).

Re:What's wrong with 1975 programming by andrewf on Thursday July 29, 2010 @03:06PM (#1558822)
I don't think Postgres' "default config" is really the "recommended config." In particular, the default size of Postgres' buffer cache is small, to reduce footprint on non-dedicated servers, and to fit within most systems' default SysV IPC limits. (God I wish they'd just use mmap()).

Postgres does expect that in addition to its explicitly-managed cache (the shared_buffers setting), the OS will be caching a bunch of stuff as well, and its query optimizer takes this into account (the effective_cache_size setting). Reasonable conservative settings are 25% and 50% of system RAM, respectively.

There's no One True Source on this AFAICT, but http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Serve... and http://www.postgresql.org/docs/current/static/runtime-config... are handy.

Re:What's wrong with 1975 programming by dedward on Wednesday July 28, 2010 @09:24PM (#1556242)
Well, with the Squid example, it's that paging out to disk wasn't even really a consideration. What if your system simply has lots of ram and no swap? THat sort of takes care of things right there, doesn't it, and let the OS file caching mechanism do it's thing. After all - if you are swapping, you'er taking a huge performance hit.