They still try to sign up, but none of them can actually bypass the captcha. I implemented a custom callback that returns a HEAD response with 404 to fool them (because I noticed that attacks intensified when I returned 2xx or 3xx responses).

THIS is what I'm talking about :) Great work! Seems to me this is very centered on Rails apps. Would it be possible to do somethink like this for "everyone"? I have nothing agains Rails, but as a js/php/python develper myself, I wouldn't know where to begin with this..

The easiest way to start would be introducing an invisible form field [0] and on submission, checking if it is not empty. Give this form field a random name so that it wouldn't be populated by password managers.

You can take it one step further and check how fast the form was submitted. If it is below the predefined threshold, it is probably a bot.

[0] To hide the form field, you can use one of these snippets, interchangeably:

    "display:none;"
    "position:absolute!important;top:-9999px;left:-9999px;"
    "position:absolute!important;height:1px;width:1px;overflow:hidden;"

It should be stated that you should either use the first approach (display none) or give a 0 height/width, so that screen readers don't pick up these fake fields and ruin accessibility on your form.

Unfortunately anything screen readers can recognize as invisible, a more sophisticated bot can too.

There's also aria-hidden to prevent screen readers from touching the honeypot field.