I was once tasked with writing a crawler for FTP servers and lost about half the hair on my head over this madness. Just when I tuned the robot to work with 10 servers, an 11th would pop up that behaved totally differently. Thankfully, I'm not in that job anymore -- I'll probably never touch FTP again.