So instead of ads you want a system where random relevant apps are floated to the top of searches? How is that helpful for anyone?

Maybe I'm missing something, but it seems like Apple is only doing what every other search system with embedded ads at the top does.

So you are saying Google search is useful only because of ads?

for many search terms, and in my opinion, yes. a typical use case on an app store is searching for something like the example given on apple's promo page here: "photo filters". neither the app store or google has enough intelligence to make a decision about what the best result is, given a query that vague and returning that many results. the most useful thing they can actually display is likely an ad.

try searching for something like "car insurance" on google with adblock turned off - the majority of the visible page area will be used by some sort of ad unit, and the native results are pretty useless.

If it were free, wouldn't everyone do it, thereby negating any benefit?

Probably because Apple can..