Very much this. I have made a point to give somewhat legitimate answers to these questions out of fear that a phone agent would ask them someday and that I could fall victim to exactly what you describe.

Phone agents don't always have to actually enter the security questions to access your account. sometimes they can simply see the answers on their screen and are able to make a judgement call. Don't trust humans, especially not humans who are incentivized to help you as quickly as possible.

Also its easier to say a word over the phone than it is to say a random string of letters numbers and symbols.