I had done "rm" of a big log file to free up space on the customer's server but our process kept on filling the log file. I assumed that the disk has got enough free space and I got busy with something else. The space was never freed up as the file descriptor was kept open by the process. Ultimately the entire disk got filled up by the opened log file and their server came to a grinding halt. I think the customer stopped using our product after that because we never heard from them again.

learning from this experience: never do an "rm" on the log file, instead do "truncate -s 0" on the log file.