As someone that used to work on recommendations systems ... the problem is that the signal is just so robust. Stuff other people like really is one of the stronger ways to predict what you will like. Most additional data sources that allow us to customize further and identify what sub-groups you might belong to and thus prioritize highlighting things of relevance to your subgroups tend to empirically show lower performance for the overall population. The only thing I've seen work is focus on the "most popular" things for the most important demographics (such as those with money to buy things) which might overfit for that demo, and underperform overall, but still performs the best on some higher metric like revenue.

I commented a while back on someone asking "Why recommend me a ladder if I just bought a ladder, couldn't the system understand that people usually only buy one ladder?" But the fact is, the commenter was wrong about his priors. Actually someone who buys a ladder is indeed more likely to buy another one ... even if most people that buy ladders only buy a single ladder. The fact that you bought a ladder, puts you in the cohort of people that buys ladders, which means you are more likely to buy another one than someone who isn't on the record buying a ladder.

> the problem is that the signal is just so robust.

That's why you aren't seeing the better solution. You're only seeing what you know, which is the (tired old) signal you know how to detect. But you already detected it, and monetised it.

A better recommendation signal would be the one that you don't know how to detect. It's the signal people get unexpectedly from other people, when they chat about the thing they just bought. Call it the n+1 recommendation signal.

> The fact that you bought a ladder, puts you in the cohort of people that buys ladders, which means you are more likely to buy another one than someone who isn't on the record buying a ladder.

OK, but that's still the n recommendation signal, which has already been monetised, and won't be refreshed for quite some time, in the same way a Venus fly trap takes time before it will open and trigger again.

What can a person do with a ladder that she couldn't do before?

That question is what begets the n+1 recommendation signal. It isn't a question that even gets asked explicitly. It happens through association.

She says "I got a new ladder, and a paintbrush and some paint." and her friend says, "Oh great, I just used my ladder to hang a bird feeder from the sycamore tree!" and then she says "Oh! I've got a sycamore tree too! Where'd you get your bird feeder?"

> But the fact is, the commenter was wrong about his priors. Actually someone who buys a ladder is indeed more likely to buy another one ... even if most people that buy ladders only buy a single ladder. The fact that you bought a ladder, puts you in the cohort of people that buys ladders, which means you are more likely to buy another one than someone who isn't on the record buying a ladder.

Be that as it may it’s still ridiculous. There are very many other things I am going to buy before I buy another ladder. And Amazon should be able to know that because of having seen me search for very many other things in the past that I have not yet bought.

And if nothing else, couldn’t they let me at least click a little button next to any suggestions to tell their algorithm that no I am not interested in this recommendation?

I'm sure the poster you're replying to knows this and wishes it weren't so.

That's because the optimization target isn't "what will let you grow best as an individual human being", but "what will make most money for the parent company". Which translates to what's most "engaging". Which translates to what's most dividing and most outrage-inducing.

That is what you get, and I thought in 2020, this would be an accepted wisdom: if you optimize for what gets the most engagement from people, you end up with things that cater to our most primal urges. It's a statistical phenomenon; you're more likely to ignore a nugget of deep knowledge than a headline which makes your eyes red before you even finish reading it.

SSC had a good essay on it back in the day: https://slatestarcodex.com/2014/12/17/the-toxoplasma-of-rage....

I wonder what would it even take for the first "what will let you grow best as an individual human being" algorithmically even possible to judge? The spurious correlations alone from bad attempts to quantify it would be amusing.

For starters, I'd give people a honest and functioning search and filtering interface. Like: "Give me videos on topic X, order by date, then viewcount, filter out the blacklist". Let people save such queries for quick access in the UI.

Give the ability to provide negative feedback. One-click "blacklist this video/channel". One-click "don't show me things like this". One-click "show me less of these". In a way that actually works.

Allow to explicitly tune the recommendation algorithm. Like, modify the list of categories that get prioritized, and even their relative priority.

All that boils down to: give people autonomy to curate their own experience to match their own goals, instead of letting an algorithm do it in the background to optimize for company's goals.

As for "the spurious correlations from bad attempts to quantify it", yeah, I'd love to read about them too. But perhaps not on the front page of the news, in context of how it hurts people.