Not necessarily - if you catch it on unsuccessful login attempts and have no evidence they got into the account, it's likely they moved onto the next email address in their list and you can still go in and delete the account pre-compromise.