I think it's mostly firefox just holding things in memory since so much is free. Eventually it just all adds up and eventually I try to compile something and then it will run out of memory.

Yes, I can go setup an oom daemon, but Linux should handle this for me out of the box (ideally with some sort of logic to avoid killing something like X taking down every single program I have open). Getting stuck for half an hour or longer is just not acceptable to let happen.