With some ISP's caching DNS for >48 hours regardless of your TTL. And sometimes browsers caching pages for even longer. Stopping offering the service is fine. Ending up with your domain pointed to a site that isn't yours is an entirely different thing. They could run a phishing site and users wouldn't have any way of knowing.