This is a natural extension of any platform geared for the general public as creators, because managing all that for someone in one place provides real benefit to the user. It's just as you note that benefit comes with loss of control.

It's really no different than anything else in life. People hire contractors to do larger home repairs, and the contractors contact and subcontract other specialty workers as needed (drywall, woodworking, plumbing, electrical) for things they can't cover themselves. As the homeowner, unless you make an effort to know who these people are, you won't know who they are and may not know what their company is. The benefit is that you don't need to know any of it and can just let the contractor coordinate it all for you.

Youtube is just like that contractor, except instead of a local guy that cares about your business it's a giant multinational which is hard to get to care if they screw something up for you because you're one of many millions of other customers.

The parallels of you can handle all this coordination youtube offers with more control are obvious. Either hire someone to do it all for you or handle it yourself if you want control, because no matter what platform you go to that handles it for you, chances are it will exhibit some or all of the problems that Youtube has, or even new ones.

I would really like a recommendation engine that I own and control.

It would know what I read and watch. It would find new stuff for me, dedupe it, listen to my feedback,and learn from its mistakes. It would work with all kinds of media, not just youtube or HN.

And it would all be private. I don't mind paying for my content but I do mind being the product.

As a computer program, this is probably not feasible. While in theory you are possibly capable of writing the algorithms for your own recommendation engine using whatever ML tools are popular today, the issue you'd have is data. YouTube's recommendation algorithm effectively only works because it's capable of aggregating millions of users behavior to feed into their own system. Getting that data personally would be very difficult both because YouTube doesn't want to enable competitors, and because consumers would be extra freaked out to find out about some random engineer outside of YT getting their watch history.

As a non-program, you can always depend on the means that humans have used for content discovery since time immemorial: word of mouth.

This is maybe a tangent, but do these ML-powered recommendation algorithms actually, like, work well for people?

I use both YouTube and Netflix pretty extensively. These companies are both held up as ML success stories. But, at least in my personal experience, they suck! My front-page feeds consist of (1) content that the platform knows I have already seen because IT IS IN MY WATCH HISTORY, (2) content that is trivially related, like by having the the same channel/director/actors, and (3) random junk I have no interest in watching and always regret clicking on.

It's not that the content isn't out there -- I'm able to find stuff I like through "big spreadsheet of all the Netflix movies" websites, online recommendations from humans, and YouTube channel crossovers/cameos. But I've never had that magical "the algorithm knows you better than you know yourself" experience that every ML engineer I've ever spoken to seems to be convinced their systems achieve.

Are all ML engineers mindless drones with exactly average movie preferences, or am I some kind of incomprehensible weirdo?

So true. The thing I find bizarre about youtube is how bad not only their recommendations but their search is. This is a Google company but somehow if I search for a specific episode in a specific series (eg one episode in a game walkthrough or whatever) it can't get the ranking right?

I also still don't understand why yt recommendations are as poor as they are. I've tried taking the time to try removing certain things from the "recommendation history" (or whatever they call it), flag recommendations I'm not interested in etc and it does improve things a bit but still it recommends things I would never watch in a million years (eg right now I have 4 Joe Rogan recommendations on my homepage even though I've literally never watched any of his things and have zero interest in ever doing so, and click on every one and say not interested etc, for a while it was obsessed with trying to get me to watch MMA content even though I can't bear watching other people hurt each other).

And the whole issue of recommending videos where it knows I've watched the whole thing, I just don't even know where to start.

On search, I wonder if this is deliberate; if they gave you exactly what you want as the top result, you'd click that one and not look any further. But by having to scroll past a few other videos, maybe you'll see another one or two that you also decide to click on, resulting in more overall views.

This is a different strategy to Google web search, where the monetisation occurs on the search results page (via ads there) rather than on click-throughs, so there isn't an incentive to try and get you to click several tangentially related links.

> random junk I have no interest in watching and always regret clicking on

if you regret clicking on it, that means you did click on it and the recommendation did it's job; remember that the job of the recommendation engine is not (directly) to suggest content you'll like, it's to suggest content that will get you to click. Ideally those goals would be aligned, but not necessarily. YouTube is simply incentivised (via ad revenue) to get you to click on as many videos as possible.

At least in my case, it's not that I'm constantly clicking on things that it turns out I dislike. Rather, it's that the few times I have clicked on a recommendation (because hey, maybe they know something I don't, maybe Sonic the Hedgehog actually is a good suggestion for people who liked Eternal Sunshine of the Spotless Mind), I've regretted it.

Right, but my point is that from YouTube's perspective it doesn't "matter" if you like or dislike the video, if they can get you to click. And clearly this strategy works on some people, or they wouldn't be doing it.

So while it may not be overly successful on you (i.e. you only click occasionally), if it works on a significant portion of people it could still be a successful strategy overall.

Sometimes you click even when you don't want to. For example on youtube on ipad it's very difficult to click on the vertical three-dot hamburger icon on a video (that you need to use to tell them you're not interested in a video) without clicking on the video by mistake. The hitbox on that icon is all wrong.

Open Data is a thing, I can imagine some PeerTube instances would publicly share their data (anonymously etc), in order to build open recommendation systems.

Personally, I would gladly donate my (anonymous etc) data for this noble goal.

Open data + federated learning (somewhat ironically a Google invention) could help with this, but it has a long way to go, and a lot of opportunities for incumbents to fight back.

The problem is (as with all open/federated systems) spam.

How do you figure out which data has been faked to drive views to particular content?

Except that youtube’s recommendation engine It terrible and getting worse. I am on youtube all the time and all it ever recommends is things I already subscribe to. And it seems to be steering further into things I have already watched.

Data offers diminishing returns, and at sufficient quantity introduces vetting and validity problems --- see Amazon's reviews crisis or fake Yelp reviews, as examples.

Data quality and relevance matter far more.

Well just for me I don't care what's trending so don't need the feed of other people's views. But absolutely needing a feed of new posts might be problem.

That's really missing the point. Recommendation engines are a lot more dumb than you are thinking. They don't understand the content at all. They only know that thousands of users that watched the same videos as you in the past also watched this other new video, so you might like it as well.

A recommendation engine that works solely based on your own watch history would require a totally different approach using algorithms that are years/decades/forever away from existing.

Good points.

It does make sense that the job is simpler with everyone's history and inputs, if you can keep PII out of it. Are there any good references on recommendation aggregation algorithms?

Here's a simple proposal that keeps everyone's stuff anonymous and allows a rec service to pay bills.

Distribute the pool of everyone's watch histories/prefs/etc using IPFS, which is a DHT with static names and some storage. User clients put the share data into IPFS and register into a public pool. The client remembers their own hash. Rec services (there can be multiple) pull everyone's sets periodically and aggregate/indexe them. Users can then retrieve a batch of new recs by sending a small fee to the service of their choice along with their hash and the server will place some new recs into their bucket. If the recs are not suitable, they can try another service.

If I send you links to 2-3 rare videos and you view them, I can probably uniquely identify you (or restrict the list of potential users to ~5 users then recognize you based on other videos you would have watched). I then have access to your full video watching history.

Even without this, I could just check videos you shared on Twitter, follow the same process, and have a decent chance of identifying you.

Anonymization is very very hard. Maybe federated learning is more promising here, keeping all your data privately stored on your own devices.

> That's really missing the point. Recommendation engines are a lot more dumb than you are thinking. They don't understand the content at all.

I think you're the one underestimating modern recommender systems, especially the ones built by advanced teams in big tech companies.

They absolutely use content features. Among other advantages, this counters the cold start problem (how will you learn about a new video if it doesn't get recommended to anyone).

Have you ever seen the torrent of garbage that is new posts on e.g. Reddit or Imgur? You only see decent posts on the ‘front page’ because some people sift through that and filter out teenagers' low-effort attempts.

YouTube has hours of video uploaded every minute. A non-centralized, context-less recommendation system would mean that you are offered those crap videos together with ones that are the standard for ‘good’ now. Just get into the long tail on any search results and see if you want those.

A non-monopolized video discovery would be splendid, only who knows how to achieve that. Perhaps it would be more realistic to do the recommendations based on the numbers of ‘retweets’ in federated networks—in a more ‘social’ style instead of views (dunno if PeerTube already offers that). Also, an open system will be incessantly gamed.

+1 I'd also pay for something like this...

Wow GPT-3 is getting good.

I agree that GPT-3 is impressive (https://philosopher.life/#Aispondence).

You might agree: "thought and discussion termination isn’t cool," right? I hope you aim to live up to your own principles. Maybe you meant to say something else to the human being on the other end of the line.

I see it's getting harder to satisfy you, by the minute.

Oh, some people say I'm an insatiable monster. I appreciate that you have taken the time to reply. If you are really here for a discussion, I'll have one. I do care about what you say and don't say (and I'll prove it to you). I do ask quite a bit of those in power (https://philosopher.life/#Power).

I know what you mean, if this is what you mean. YouTube has become so laser-focused that it's become ridiculous. I watch videos and get suggested similar videos. Eventually I'm only watching one type of video.

>I don't mind paying for my content but I do mind being the product.

This is another thing that bothers me. Being guilt-tripped for not watching an ad, or being a Patreon subscriber, or YouTube Red subscriber. Even if I don't watch ads or pay a fee I am, me, this individual is valuable just for who I am. My data, my habits are valuable to Google just as much as it is for me to pay a fee or to watch ads.

Google is not the person who created the content you are consuming. That person or group of people gets nothing from viewers who "don't watch ads or pay a fee."

> And it would all be private. I don't mind paying for my content but I do mind being the product.

Some portions of a personalized, cross-service recommendation engine are a lot more viable if centralized (e.g. any scraping/indexing, and plausibly the personalization itself). Supposing such a service had no intention of you ever personally being the product, what would they have to do in order for you to still be comfortable handing that data over to them?

Host in my closet or basement with a one-way data flow. In only, not out.

How much of the general public is going to be capable to set up and maintain a server in their basement? To challenge YouTube, targetting just the programming crowd isn't going to be sufficient.

The general public seems to be capable of setting up and maintaining a Nest thermostat. The only difference would be that data only goes down, not up.

I guess if you sell a pre-constructed device, that could work. There would be other issues (you have to spend money on hardware, potentially have to get it shipped, may not be able to get access to the device at all based on your region, keeping it always on and accessible, etc).

What do you think of https://findka.com/

No affiliation, but it seems to be trying to do what you want

That‘s why you plug your twitter and maybe other socials (mailing list as well I guess, though less usual) at every possible opportunity. It‘s incredibly important to cultivate the relationship with your audience.

In the event that you have to change platform for whatever reason, you still have a line of communication.

The first "big" band I ever saw go multi-channel to connect with fans was Dave Matthews Band [1] In March of 2009, the band went from a blog-looking thing [2] that collected email addresses to putting every social channel front and center. [3]

By November 2010, they had iTunes Ping, Twitter, an iPhone app, Facebook, Myspace, the bassist's twitter feed all being pushed in huge tiles.

At the time this was extraordinary, since many artists did not have a handle on social at all and definitely were not pushing people "away" from direct connection like this.

I believe the idea was to allow many platforms to get between you and your fans/followers, but to allow the fans to choose which platform that would be.

As the OP suggests, this can be a lot of work.

[1] Coran Capshaw, the band's manager created an early dial-up ISP company in Charlottesville that went on to become MusicToday. was responsible for major tech advances in live entertainment

[2] https://web.archive.org/web/20090219153059/http://www.davema...

[3] https://web.archive.org/web/20101105102023/http://www.davema...

> The first "big" band I ever saw go multi-channel to connect with fans was Dave Matthews Band

Judged.

However, on a more serious note, I new people who were -very- attached to bands/musicians based on interactions they had on MySpace and LiveJournal.

One problem with any online exposure in the modern web is that the bar from 'unknown' to 'trolls flooding your inbox' is pretty low. I do wonder however whether small time performers wind up trapped by this: When they are first growing they probably can't pick the ideal "Social Media" person, which may limit certain exposure and opportunities.

> Judged

You don’t know the half of it. I ran a DMB blog and large online community. It was where I did a lot of my first web programming. :|

Whatever you think of their music, DMB and their fans pioneered a bunch of areas.

One of the most forward decisions by the band was to allow online distribution of their live recordings.

Being mainstream at that point meant it pushed torrenting of the live shows into the workflows of many people who wouldn’t have otherwise have been exposed to the tech.

It also drew the hackers of sorts to their scene who would illicitly capture and record IEM streams, mix them with mic recordings then combine them with stealth monopod videos to release bootleg copies of entire shows.

Only the Radiohead fans came close to the amount of enthusiasm in the DMB scene back then.

That said—if I could go back I’d pick Radiohead to follow.

Twitter has the same problem as YouTube. Email is a federated open source protocol and far superior as way of securing your fan base. It's probably a great idea for content providers to self-host a website with their videos on it as well.

I agree generally, but most people I know barely ever check their personal emails, pretty much just enough to sign up for twitter/instagram/snapchat/whatever's popular now.

In what kind of bubble do you live? Most people I know check their personal e-mails quite regularly.

Mostly people aged 18-25.

Ok, then it makes sense. That age group is still considered very young adults, and most are still studying and not even exposed to full time jobs yet. As soon as you receive important emails from banks etc you start checking your personal e-mail regularly.

That works for very niche streamers/creators. Maybe with an older audience as well. If your a streamer on twitch or YouTube chances are your audience isn‘t older than 30. I personally don‘t really use email for actual communication. Twitter is pretty much perfect to communicate to an audience. Yes it isn‘t federated but at least you‘re using a bunch of different platforms to diversify risk.

Generally I would advise new content creators to be present on every possible platform.

I don't use Twitter but I tried it a few years back. It's hard to follow relevant content if half of what you see are people's re-tweets. With e-mail I can easily sift through and find what is relevant to me.

Remember when people browsed pages to find new content by creators instead of being spoonfed by an algorithm?

I do, but the memory is well-faded, and growing dimmer...

Yeah I remember when no one but nerds like us used the web

I disagree somewhat - well, I think simply backing up content is insufficient but if you diversify your platform now and manage to snag a significant number of viewers then you should be good. Shared platforms like DLive (a la pewdiepie fame and others) or watchnebula.com is one way to do this and can be very strong for topically related subjects - independent platforms like Dropout.tv are also an option but you're either hoping to win the bandwidth vs. advertiser cost balance or going to need to charge for access - both of which can lower your subscriber rate.

I think it's hard to make a fully independent platform so I think topical ones are a better option - but YouTube is really in a position of power due to the network effect. As long as they don't shoot themselves in the foot with controversy and banning then they should be able to easily hold onto their position - everything challenging them has an uphill battle.

Yes, unless you crosspost to Instagram or Facebook and tell people in your videos about all three. Or is that frowned upon at YouTube and/or monitored if you have a high viewership channel?

YouTube doesn't care about that.