Fundamentally it's a question of fingerprinting the behaviours of humans versus bots. The problem is that it's becoming increasingly difficult to distinguish them, particularly when bots are running headless chrome or similar, and real users are automating their sign-ins with password managers.

I don't do much of this sort of thing, but numerous things come to mind. Aim to identify and whitelist obviously human browsers, blacklist obviously robot browsers, and mildly inconvenience/challenge the rest.

For example, an obvious property of a real human browser is that it had been used to log in successfully in the past. Proving that is left as an exercise for the reader, though it inevitably requires some state/memory on the server side.