The bigger worry would be if they removed some IP addresses from their list. Those IP addresses would be juicy targets for hackers to scoop up and attempt an attack knowing that people have whitelisted them and that they allow access to what is likely relatively poorly protected infrastructure.