It was stopping the attacks - it was just that the attacker would try 10 password attempts, then get blocked by the plugin and trigger the alert message. Then the attacker would switch IPs and try 10 more. One morning they had gotten a ton of messages and I found about 250k login attempts in the security logs. So the plugin was doing it's job, but it's better now that the attacks don't even make it that far. In fact you can't even hit a page within the wp-admin folder which is nice in case some type of zero-day exploit surfaces on a file within that area.