But yeah, at the end of the day, someone who spends a lot of time making cryptographically secure passwords and rotates them every six months is probably less secure on the balance than someone who uses 4-digit pins they rotate every three months.
If lists of usernames/emails are available elsewhere for users of the site - a NastyPerson can reasonably quickly lock your entire user base out of the system (either deliberately, or by a poorly coded password attacking bot).
If you have per-IP lockouts then you're still open to bot-nets and the problem doesn't get much better.
What seems to work best is to slow logins down rather than lock them out completely. Still lets users in, but slows down bulk attacks enough so that the risk is low.
And having nice metrics / reporting tools so that you can spot attacks as they're happening and make appropriate case-by-case responses.
Forcing a CAPTCHA for a new IP address or when a "you-have-logged-on-from-this-browser-before" cookie added to a lock-out would definitely be an improvement.
1) Username is not known. No reason the attacker would know this short of a leak / social profile with identical username. In this case, submitting random usernames and getting a hit should trigger a lockout and corresponding alert.
2) Username is known through system compromise. In this case, the account should be locked out and a new password generated just in case.
So I don't really see causing a deliberate account lockout as a primary attack vector other than trying to exploit weaknesses in password reset processes across multiple sites (like what happened to Dan Harmon).
Therefore, I would say that having a lockout mechanism is better than not having one.
You're assuming that the lockout is aimed at an individual. In my experience it's aimed at the site. They're trying to lock out N% of users. In which case going for common names, email addresses, etc. will get you a surprisingly large chunk of the user base.
And that's ignoring the many sites where usernames are public, or easily accessible by a "normal" user that can be used to scrape them for the DOS/password bot.
Therefore, I would say that having a lockout mechanism is better than not having one.
Since I've seen DOS attacks that use them - I'd tend to disagree :-)
Unless you're in an environment that makes them effective.
For example my bank uses a lockout - but it has a private customer number that's not visible to others, along with a separate password and a 8 digit PIN (of which it only asks for three random digits). It's only when two of the three bits of information are correct that the lockout count starts - making bulk lockouts for the site pretty much impossible. So there it's an effective tool to prevent password cracking.
There will be people who say that it would all be fine if they just used bcrypt, but the fundamental problem is that if you don't use different passwords on every site, you're hugely at risk.
Consider a situation where you've given your "trusted" password to a site that you know does all the right hashing with password storage. Your super password is 20 characters of alphanums with specials and you're feeling pretty smug. You now hear that the site was hacked, but you're not worried as all they can get is the hash and let them just try brute-force, right?
Wrong. They hacked the site. They changed the login form and grabbed your password in plaintext. A hack means that they compromise part of the infrastructure. At this time it's sufficient to just grab database data. Changing presented HTML is possibly a little harder as it needs to stay up for longer, but if someone has access to the database, they probably have access to a whole lot more. This is why lastpass.com is less secure than storing your password database yourself.
TL;DR: make your passwords different.