That would make total sense if the human mating ritual involved the male impressing the female with the size of his testicles.

How do you know it didn't? Culture changes faster than evolution.

http://mentalfloss.com/article/79897/why-arent-classical-sta...

“Ancient Greece was a highly masculinist culture,” she explained. “They favored ‘small and taut’ genitals, as opposed to big sex organs, to show male self-control in matters of sexuality."

exactly, most of culture changes fast and for there to be a sexually selective pressure on testicles for a long enough time to make a difference, it would surely persist into recorded history. Either human mating practices are cultural and transitive so they would not exert continuous evolutionary pressure, or they are somehow more ingrained and thus could exist long enough to trigger noticeable evolutionary changes.