In arbitrage's defense, absence of evidence is frequently not evidence of absence. This is just a particularly poor choice of place to cite the principle.

We are left to wonder why "of Babylon" ever got attached to the expression. My reading suggests that about everything out "that way" was often referred to Babylon, much as later (and still!) everything from the Dardanelles east was "Asia", even though Asia strictly included only what is now westernmost and extreme northern Turkey.

We might also wonder why writers of the time did not carry on about Nineveh and its wonderful Garden. My reading suggests that writers of its time preferred to dwell, when they mentioned neo-Assyrians, on their military aggressions and impalements. I speculate that it might have seemed confusing to also mention gardening.