Usually when the problem gets really bad, memory use has jumped into a few hundred MB. That seems to happen when I've switched between local login and acceding the machine via RDC.