The only downside is you don't have remote-control over the depth/aperture. you'll notice most of the high-quality editorial photographs have great depth that isolates the animal and makes a greater image.
with this setup you need a really wide lens or a fish-eye to make a good composition. cool up-close photos but you give up control over the mechanics of the camera.
Or maybe it's just the "I could have thought of that!" feeling.