The dark side of the Earth is very bright in IR. Getting far away from the Earth and moon makes them dimmer. Also, going for L2 makes sure the Earth, moon, and Sun are all always in the same direction, so the sun shield will block all significant IR sources.