Couple things here:

First, ecosystems don't work on this concept of "balance" or "problems to fix". There are ecological niches that are exploited by a species or not. We have both generalists and specialists, predators that will eat anything and predators that have specialized in a particular prey or hunting approach. The world is full of organisms with remarkably complex behaviors, the field of evolutionary biology would collapse if this sort of behavior disproved it.

Second, there's really nothing hand-wavy about how specialized behaviors evolve, and in fact there is a mountain of research in the fields of evolutionary biology and behavioral ecology that have significant explanatory power here. I would argue that you're the one being hand-wavy saying "I don't think a simple gradient descent...". Evolution is not a simple gradient descent, and no evolutionary biologist would ever argue that it is.

It might be more helpful to think of it as small changes in “genome-vector” taking place across generations of the species with the filter of it being not bad enough to cause extinction of the bespoke variation, as opposed to evolving towards more fitness as an optimisation objective. When thought of like this, one can imagine how high-dimensional the “feature”-space becomes, which leads to such intricate engineering we see in nature.

There's also typically a ton of standing genetic variation that is relatively fitness-neutral, but can then lead to more rapid or directional evolution if conditions change. For example, standing variation in silk elasticity may have been more fitness neutral when capturing other prey, but after encountering this ant species, spiders with more elastic silk were more successful and had a fitness advantage.

> standing genetic variation

An interesting illustration of this might be variety seen in dogs, which are still one species that can reproduce together.

Presumably there was some ur-mutt, where almost all the variations we see today were already there, latent, until humans nudged the balances and blends.

Attempts to systematically domesticate foxes [0] suggest that certain packages of physical traits come along with selecting for calm or friendly behavior.

https://www.pbs.org/newshour/science/domesticated-foxes-gene...