Yeah, but I have the advantage of knowing all the forms of the questions. If you were to change one word then mine would break. Or misspell anything.

I just typed in similar questions without following the format and it performed poorly. But at least it's a good proof-of-concept! :)

Name the third item in the list: book, fork and spoon.

A: FAILURE

Q: What is two times 8?

A:

Which is a number: yellow, seven, book, flag?

A: FAILURE

They don't give questions formatted in other ways, so I didn't include that in the logic. As someone else pointed out, even deleting the question mark would break it. (However, making the question mark optional would be a two second fix.)

Sorry, but what do you mean by "knowing all the forms of the questions"? It seems like maybe you mean all that all the questions formatted via one of several formulas like: name <x> of list<a..z>. Which would imply your tool simply identifies the format then applies some specific logic to solve the question. I'm unclear that if that is what you mean, why then do you know all the forms of the questions? Text capture purports over 180 million questions, so isn't it likely you may not have found all the possible formats of the questions? Or perhaps did you know because you have a relationship with the text captcha creator and they provided that info to you. Or better yet, did your comment mean something entirely different.

Just curious, thanks.

You are correct, I look for a pattern and apply very very basic logic.

I just got 1,000 questions from the demo page and looked how they were similar. For example, there are many questions of the form "[list of something]: the [ordinal number] [type] is?" so I put that as a rule in the grammar file. I did this for about twenty different types of questions, each of which has three or so different phrasings.

And yes, they may have 180 million questions, but in reality they have only a hundred or so modulo the specific words they pick (for the items in the list, or for the day of the week, or for the letter that the word starts with).

  > I put that as a rule in the grammar file

What are you using the process the grammar file? What type of grammar?

I wrote a parser generator, so I'm using that.

It's BNF-like.

It's basically an arms race. They could inject a few misspellings and random noise here and there to throw your system off.

Fuzzy pattern matching would probably solve that.

Great job! I wrote up that post about WolframAlpha and wonder if there's an API so you could integrate WA to make it more robust?

Right now there's no API or anything -- I wrote this just to see if I could. I'm not sure if I'll be putting up an API, because I figure by the time I'd do that I can just release the source for it. But if you really wanted I probably could provide an API it in a couple of days.

The world orange has which letter in the penultimate position? A: FAILURE

Aircraft fly through the? A: FAILURE

Q: A beetle has how many legs? A: a

It does well when it recognizes the questions, but there's still plenty of room for writing new ones. It does show that it's an arms race, as someone else noted.

Very true. The person writing the questions has the advantage in this one because they can always throw in more variations and I would have to figure out all of the new ones.

There is, however, an inherit limitation to question-asking, and that is it must be understandable to a general population. So I would switch from using a parser to just looking for key words and then matching from there. (Which is, for example, how I find names in a list.)

Even if they made it so I only get one fifth the number of correct answers, a 20% success ratio would be high enough for concern. The attackers can just keep trying and trying and if 20% of the time they can register a new bot, then they win. The defender must win a much larger percent of the time.

First question could be "air" or "sky". I would guess 6 or 8 to the second one.

I would probably fail both, so I guess this isn't the kind of question that would realistically appear.