I thought it was also about usury not being allowed (for Christians, Jews, or Muslims) intrareligiously but only extrareligiously, i.e. Jews could loan money to Christians, but Christians couldn't loan money to Christians.

Yes. That's the usual story indeed. I'm not sure where the concept that banking was 'dirty' came from - I suspect that's mixing up banking with lending. Lending at interest was subject to religious prohibition for all but Jews who interpreted the Torah in a way that forbade lending only to other Jews. Thus Jewish people were not "forced" into banking because they were banned from agriculture (lol, how would that even work?) but rather, became bankers by default because it was highly profitable and others wouldn't do it for religious reasons.

This is the main reason for the ancient historical stereotypes linking Judaism and money/wealth/power/etc. And why names like Goldstein (gold stone) are considered Jewish names.