Actively cooled IR sensors can achieve higher resolution and accuracy, but ambient temperature IR sensors have been around for a while. As long as the incident IR can drive the sensor element away from ambient temperature, then you can have sensing.

IR sensors don't work by detecting heat changes. They convert infrared photons directly to electrical current.

Now I don't actually know jack about IR sensors, but I would assume that increased temperature means increased noise, as it does with visible-light CCDs. People refrigerate visible-light CCDs too, for taking low-noise astronomical photos.