This is correct. Lets say you have an ecom site with product search. How do you know if the search is good? You search for stuff and compare to what you expect.

Even if it sucks, as long as their is revenue lift, you will keep it until the next solution comes along.

And Algolia has great analytics, so you can actually mesure business value from actual search queries: you can tie a query to a purchase and then run further analysis on that. It’s powerful and doesn’t require any exceptional engineering talent to use.

Totally false. This is like massively overfitting a high-order polynomial regression to your data. The fit looks good enough, then the next data point comes in and breaks in a way the existing model cannot be hacked to account for.

The search results you believed were implicitly tuned to some feedback mechanism slowly experience creep as the customer cohort changes and data distribution changes until before you knew it your management of the search solution is a ceaseless game of whack-a-mole siphoning off engineering resources at a rapidly increasing rate. It’s the same false promise of just having some engineers stand up Elastic Search.

Not all decisions are good decisions, it depends on who makes the call. In most cases, someone ask you to use a solution because it looks/feels better. In this case Algolia showed how fast and how well it could be implemented. Once the person who takes the decision is convinced it will be implemented. It's mostly marketing. Probably less than 1% of all e-commerce websites measures the impact of a decision.

I did say as much in my original post: Algolia and Rekognition are marketed at CTOs and directors of engineering who want to be sold on a magical line item that removes a whole concept area from their concern, especially one associated with the difficulty of hiring and affording good machine learning staff who can work on the problem both pragmatically and theoretically. They want to be sold a story.

I will say though that your 1% claim is way off in my experience (which includes 3 medium and large ecommerce companies). These companies employ armies of product managers and analytics staff that measure the shit out of everything from the color of a button to the size of font in a banner display for a discount promo code. These things aren’t usually measured because they find value, rather just to give the appearance of data driven decision making and justify job perpetuity.