What I would personally love is an aggregator, or many aggregators, that act like YouTube, but are actually a distributed network of video sources.
The key to the concept is thinking of a source like a library of content, and the individual uploader manages their own source. You could consider it a channel, and then that uploader can syndicate their source/channel to any aggregator they want. Sources can be private/public, sources can be tagged, discoverable, and searchable.
Another key point is that uploaders need not host things themselves, SaaS products will quickly fill that role.
a) Anyone can start an aggregator and immediately have content, lowering the bar for entry.
b) A niche aggregator can pick or have submitted to them video library sources that match their niche, and can curate them just like any other video hosting site.
c) A cottage industry for video hosting can utilize an "average user" friendly UI to allow for anyone to create a source and submit it to these aggregators.
d) Allows free information to flow freely
f) Allows content creators full control over deals with aggregators on how they'll be reimbursed for access to their source. You can imagine YouTube accepting sources with an ad revenue share model just like they have now. Some aggregators will choose to "cache and redistribute" content from sources to allow for consistent UX to their users, but that's their decision.
Which will, of course, drive all the normies away. (Well, okay, all the pirated content might not, but then you'll also include 'all the media companies' and 'the government' as your list of enemies.)
And if you add federated moderation (Like Mastodon did), you're going to quickly understand why federated moderation is a labour-sucking, drama-ridden, centralized-censorship, tire pyre.[1] (Like Mastodon did.)
[1] Mastodon instance administrators blacklist instances that produce problem content and problem users.[2] (Child pornography, nazis, furries, etc.) These ban-lists are centralized, because nobody has time to investigate every single other instance in the network. I hope you trust the person running those centralized lists!
[2] Oh, and if a troll group does not like a particular person, they can follow him around, shitting on every instance that he posts in. How do instance operators deal with this sort of thing? By banning[3] the target of the trolls, because then they can stop wasting time playing whack-a-mole.
[3] https://www.theverge.com/2018/8/31/17801404/mastodon-harassm...
The aggregator can build their site however they want. So then the moderation and community guidance is their responsibility. Maybe the site doesn't have comments, maybe the site has a 30 day source verification timeframe. Unsurprisingly I don't have the solution to the internet's cultural problems, except to say that moderation is probably the hardest problem for any internet community, and YouTube hasn't figured it out yet either.
In some countries, the hosting services would also be on the hook for policing of content on their hosting. Not everywhere has safe harbour type laws, and some authorities have the ability to take down servers they deem break the law.
Of course, an aggregator could accept all public sources, and inevitably succumb to the unmoderated internet.
https://nolanlawson.com/2018/08/31/mastodon-and-the-challeng...
https://hackernoon.com/mastodon-is-dead-in-the-water-888c10e... [1]
https://www.theverge.com/2019/7/12/20691957/mastodon-decentr... [2]
[1] See section on identity and moderation - you can't decouple them when you have a hybrid between a content platform and a social media application (Which is what any video hosting service that allows comments, or shows recommended videos, or cross-links to other distributed aggregators is.)
[2] Bad[3] people show up to the community, everyone loses their mind, hard-coded bans starts happening, because the community interfaces with things like app stores, and app stores have policies that don't care one whit about your federated free-as-in-speech ideals, not to mention that developers and moderators are human beings too, and maybe they personally can't stand allowing furries/commies/fascists/homosexuals/homophobes/Englishmen in their space.
[3] For whatever someone's definition of 'bad' is.
I say this in good faith, as I'm not part of the alt-right nor do I agree in any way with them. But toxic people are of all kinds and we should be aware of those in our camp who do harm to our causes by acting in deleterious ways.
--
However, it raises a more valid point about 'what do we do about controversial figures' in general, and you would expect, as with most federated services, to be able to prevent "non-local" personas to be blocked from interacting with local users.
In those situations though it degrades the experience; the best part about mastadon though is that you get to choose who you federate with, so you could, in theory, have linked mastadon installations for government officials (who are often controversial) and signup could be linked with an oauth2 identity system that ties back to a real identity somehow (this would be possible in Estonia with their e-ID systems).
He will denigrate, attempt to belittle and often pick very bad faith arguments about the smallest of issues.
To give an example of his demenour: https://youtu.be/pHckKckhBYc
There's also this account from a fan, which goes into how he is: https://v1.escapistmagazine.com/forums/read/18.874144-I-need...
He also called PewDiePie "A Racist Piece of Shit Bigot" and while PewDiePie did use a racial epithet and there's no excuse for that, he did apologise profusely and regardless that's not a decent way to engage a stranger. https://boundingintocomics.com/2019/04/02/wil-wheaton-attack...
He will block you for any disagreement; and I don't mean people being crappy to him, anything that goes against what he says: https://steemit.com/blocking/@awgonnerman/is-this-why-wil-wh...
For whatever it's worth his introduction to mastadon was mired in controversy after he bleated calls to have a user blacklisted from the platform for the following joke:
> “Hey, did you watch that awesome show on Bofa last night?”
> “What’s Bofa?”
> points to crotch “Bofa deez nuts down here.”
And when Mastadon wouldn't (couldn't) blacklist the user from the platform he got viciously angry.
EDIT: His fans are on hackernews; please don't "downvote to disagree"; he is known to be toxic and curates blocklists of anyone who slights him in any way. If you disagree, please prove me wrong.
(It's 95% ready but still a WIP, so excuse any clunkiness and the lack of content)
It's a content aggregator for content creators (plus custom postings) -- Make your YouTube channel, Vimeo Vids, Twitch streams, etc. available to your audience all in one spot.*
My hopes is that once this catches on, it will become easier for new video hosting services, etc. to pop up since the huge current barrier to entry--providing an audience for content creators--is no longer a necessity. The only thing new services need to compete on it the value they contribute as a content host.
This isn't quite as technical a solution as what you described, but the current problem is that of coupling content hosting and audiences, and I believe severing that connection will bring about many improvements to online content creation and distribution.
Feel free to respond with any questions or email me (yaniv <at> mynexus <dot> io).
*The list was about to include Mixer too, but that opportunity faded away, unfortunately. Will be adding more content sources over time as they are requested by creators.
I've been thinking the same thing about Amazon lately: things would be a whole lot better for the experience of buying things "on Amazon" if "Amazon.com, the storefront website" and "Amazon.com, the platform+marketplace that sellers register with" were two separate businesses.
I'd love to use the latter without the former—to use some third-party "Amazon client" that talks to Amazon's marketplace backend but has its own way of searching, browsing, and recommending products, with its own curation, its own reviews, its own ratings, etc. added on top. In the end, you'd still be putting Amazon.com warehouse SKUs into your Amazon.com shopping cart, but those SKUs might be ones Amazon.com the website would never have surfaced for you no matter how much you searched.
For just one thing, someone could build a "classic Amazon" frontend that filters out SKUs coming from "Amazon Marketplace" sellers.
For another, someone could build a version that applies actual modern spam/scam-filtering techniques to the SKU database, paring it down such that any random thing ordered from the "safe" subset should actually be what it seems from the listing. (Not the approach that most benefits you if you're Amazon.com-the-website, because its listings might look empty; but certainly a niche to be filled!)
Building a tracker/aggregator (like PeerTube etc) is straight forward. It's actually getting video content to clients that is challenging. Video is both large and sensitive to internal ordering. You need packet 1 before you can do anything with packet 2, if you have those but packet 3 is stalled you can't go forward.
To serve video you need enough online storage to actually house it but then enough upstream bandwidth (and good latency) to serve it to clients in real-time. Serving up HD streams at a reasonable bitrate will quickly saturate a gigabit connection.
A network of HTTP servers or even an overlay network like IPFS don't help solve these problems. Video needs a lot of storage and is very sensitive to a server's upstream capacity.
Torrent based streaming can distribute that load but introduces its own problems. Torrent based streaming clients just request video content blocks in sequential order versus normal BT clients just accepting a random ordering of blocks. But again unless you've got a bunch of high bandwidth seeders it's difficult to serve content at playable rates. If the content is encoded to use 5Mb/s in order to play it back without buffering you need it delivered at least that rate.
Both methods to distribute video suffer from popularity problems. If transitional servers stop hosting old/unpopular content or seeders for it drop off it becomes unavailable on the network even if it's still referenced in the aggregator.
Not that YouTube is an overall good service or that monopolies in the space are good but online video is an area that gravitates towards centralization if not monopolies. If you want to compete with YouTube, or more likely Vimeo or another smaller player, you'd need to replicate their infrastructure. That's the difficult/expensive thing to build and maintain. Other self-hosted services are trivial by comparison.
I'd love to set up a seedbox that works like a ~bitcoin mining rig but consumes upload bandwidth rather than CPU/gpu time. However I'm not sure anyone has figured out a good way to do this while avoiding copyright issues.
End users do not want to deal with micropayments, they'll just watch one of the billion hours of content on YouTube. Few content creators have any sort of following that will follow them to SelfHostedTube.
Self hosted video streaming (of any configuration) is competing with "free" and convenient YouTube. Unless some independent service offers a similar level of service it's not going to catch on.
You're right though of course, which is why we have a YouTube monopoly in the first place.
Video distribution is expensive and technically complicated.
Say I've got a video channel and most of my content is "talking head" style. Encoding 1080p using in a widely compatible h.264 configuration (Main profile level 3) I could easily get good quality output at a data rate between 1-2Mb/s, let's say 1.5Mb/s. That's about 11MB per minute of video for storage purposes (for that single config).
In order to serve those videos to streaming clients I'll need 1.5Mb/s of bandwidth for each client. Any lower of a transfer rate and the playback will stall. Faster data rates can allow for a larger buffer but the file's data rate is your transfer minimum to actually play the content as it downloads.
So back to our video channel. Self hosting on my residential connection means I'm limited by my shitty upstream bandwidth and throttling from my ISP if not outright T&C violations. My home connection I could serve maybe a half dozen clients simultaneously. So long as my content is unpopular that might be ok. Bubbling up on an aggregator or just making an interesting video would DDoS me and no one would get to see it. It I want to spend ~$60 a year on cloud hosting I could serve a lot more connections. Then I'm out $60 a year or more if I want to make a longer history of my content available, remember 11MB a minute.
If I want to have better compatibility and broader reach I would need to encode a couple more versions of my videos at the cost of storage and my time.
So self hosting either costs me money or severely limits the availability of my content. P2P distribution can help a little but it has its own issues. Even an aggregator offering paid hosting doesn't really solve the issue because I'm still out some money. Were I to make any money from advertising or patronage that hosting cost would come right off the top.
The situation is even more difficult if you consider mobile access which is a huge portion of online video access. For a lot of reasons self hosting from mobile devices isn't practical. P2P from a mobile device is likewise impractical. So for anyone with a mobile device as their primary computing device, a non-trivial number of people, they would have to pay for some third party hosting without a meaningful self hosting option.
Demand from mobile devices also significantly increases your need for multiple encodings of a video. Mobile streaming needs to respond to much more drastic traffic conditions than your laptop on your home WiFi. This increases storage requirements meaning less historic content or increased storage costs.
TrainVids.com doesn't get many hits so for now they rely on the source' perfomance, but CatVids.com gets millions of hits and they long ago invested in a redistribution layer. Meanwhile ContentCreatorMatt pays $5/m on his source hosting with burst protection should a video go viral. Hosting can afford this because they have thousands of creators who never get a hit video. While CreatorTom only submits to large platforms with redistribution so he doesn't have to pay for traffic.
I wouldn't use P2P nor expect people to host the videos at home because that would never work for the reasons you outline. But think of live streaming and how that distribution works and that is just one possible method an aggregator could choose to increase performance of slow sources.
The idea is to break up the video platform monoculture and make the space competitive again, so that there will be some variance in performance to end users depending on the aggregator they visit isn't an issue.
There probably are ways to keep a catbirds.com running but as the difficulty increases the likelihood of the site's maintainers keeping with it decreases significantly. If the site tanks all of its hoisted content goes with it. Ask all the people that had their content wiped from MP3.com how that feels.
So from a content creator's point of view, why hitch their wagon to catvids.com? It costs them money and isn't guaranteed to be around in a year. They're also far less likely to make any money no matter how much traffic aggregators send them. Sticking with YouTube is way less of a hassle and gives them a far better chance to make money if nothing else to break even to keep making content.
The aggregators don't own or host the content in the system I am thinking of, they just consume and redistribute their sources. So if catvids.com goes away the creator's source is still online. It's also probably syndicated to a bunch of other cat video sites simultaneously from that one source. Thinking through it more, it's pretty much an RSS feed but for videos. In fact I would probably just use the RSS protocol if I were ever to try and get something like this off the ground. Have an open source aggregator template so it's easy to start one, and then monetize it by starting the first video source hosting as a service company.
I don't think it could ever take off though, the "Next YouTube" will be another massive corporate, closed platform with deep pockets or venture capital.
The other part of avoiding peer to peer is to ensure there's money in it for hosting and advertising. I hate to be cynical, but for companies to play along and join in, there has to be money in it.
Data storage costs at around $1/TB/Mo raw ($3 for 3x redundancy), bandwidth costs at around $1 / TB egress. Pricing comes from an open marketplace, and we do a lot of sophisticated stuff to make sure it's high reliability.
We've got video buffering solved as well. The latest builds of Sia (decentralized storage network) can retrieve files at about 700mbps with a time-to-first-byte of under 200ms. We haven't fully rolled this upgrade out to siasky.net yet, but it should be there in the next 1-2 weeks.
The decentralized cloud is a lot closer than people realize, the infrastructure is finally reaching a point of being competitive.
Current rates on the network are $0.70 per TB, cloud to user. Many of the nodes are serving bandwidth from unlimited connections, either at home or from cloud providers like Hetzner.
Having users act as nodes that always store and share some chunk of content is a noble idea and a piece of the puzzle, but the scale required to support something like YouTube is very different than how simple it usually is to find a torrent of a popular recent film or whatever. Ensuring that the network also always has available an unpopular 6-hour video someone uploaded last year of some birds in their backyard is a lot harder. If users can't use your video platform without some basic guarantee that the video will even exist or have peers, this would likely be a much bigger problem than most people have with YouTube (though perhaps not philosophically).
I personally feel that decentralization and shared distribution is likely the only way a viable alternative will be able to surface, survive, and not make the same mistakes, but we aren't there yet, and PeerTube is too simplistic of a solution.
One potential piece of the puzzle that I'm closely following and casually involved with is VideoCoin (https://videocoin.io/) which distributes the expensive work of encoding videos across nodes, and aims to fully power the delivery of their livestreaming product the same way. (Yes, I know anything "blockchain" in 2020 is likely to force your eyes to roll, but it's actually a fantastic product and has made lots of progress.)
I'm not sure I follow. Peertube works without peers. I maintain an instance for my personal videos, most of them have maybe 5-6 views total, and they'll still work fine whenever anyone visits the server. The peers are only there to handle bandwidth spikes and scaling.
And if you're worried that my server will go down, other instances can just mirror it.
Of course, if I stop hosting the content and nobody else mirrors it, it will disappear. But from my understanding of VideoCoin, nothing seems to be different there at first glance. If I stop paying for people to host my video on VideoCoin, are random nodes just going to keep storing it for free?
I'm trying to think of any distributed storage system that actually solves the problem of unknown content disappearing. Filecoin & IPFS don't remove the need to pay for storage, torrents still need to be seeded. The point isn't to make it impossible for content to vanish, the point is to make it easy for people who do care to archive and share the content that they care about.
I did not mean to imply that VideoCoin solves this issue, just that it is, in my opinion, a clever and functional solution to one or two pieces of this very large puzzle.
> The point isn't to make it impossible for content to vanish, the point is to make it easy for people who do care to archive and share the content that they care about.
My personal opinion is that anything truly viable and useful will need to do both, at least to the extent that this can be done. I have a huge problem with content availability being so dependent on popularity (and likely also temporal factors) and don't really think a system that doesn't inherently support content longevity (whether peers like the content or not) could be considering a big improvement over YouTube or centralized options.
If I had my way, I think a hybrid approach of some sort would be the place to start -- let peers explicitly choose to mirror and help distribute content they care about, but also force all nodes to also devote some % of whatever they're allocating to the platform to mirroring anything the network has chosen to give them after various calculations intended to maximize data integrity and performance.
Right now I think of any video on youtube as temporary and archive any videos I think there's any chance of wanting to watch later using youtube-dl.
@feross
Actually not relying on one is more portable across isolated networks, and requires way less computing resources. Plus, blockchains are usually a privacy issue.
I'm highly interested in the topics of decentralized (federated or distributed) platforms and identities, but I don't see what a single blockchain brings to the table? I could see profiles-as-blockchains, though, or instances-as-blockchains. But the rationale isn't clearly explained in the website, so it feels more of a buzzword and less of something that efficiently solves an issue (content discovery and global indexing, I guess).
Lastly, would you mind stating your affiliation? Thanks :)
They are attempting to decentralize and distribute video encode/delivery, and a big piece of that is a poof of work implementation to incentivize and reward reliable nodes.
Just one example, but there are others, and a lot of progress has been made to reduce the strain caused by throwing crypto into something that fundamentally needs to be very fast and lightweight to be usable.
Edit: Corrected duplicate links
* Storage costs are likely paid for by the uploader. Most regular uploader do (or should) keep at least a local copy of their work. Plus the original materials. So the weight of the video is likely not that significant.
* With webtorrent, the more a video is watched, the more it is shared. If you want to handle the worst case though, you'd need enough bandwidth to serve a couple times every file over simultaneously. That gets worse the more videos you have, but simultaneous watching of every one of your videos is unlikely, I think, over a certain number. It would be nice to get some stats on this.
* That's not really needed, is it? Latency isn't likely to be big enough to cause issues, and I would think available bandwidth doesn't change that much with distance. With a couple of local torrent peers, that also gets better.
Now, this isn't perfect, but there are ways this can be improved:
* Peertube intances can mirror content to provide at least one peer (and a backup HTTP source I think). It would be nice if subscribers could choose to support their favorite channels by asking their instance to mirror that content (with a lower priority than local content, of course). That could more or less solve the local CDN issue, if "peertubers" participate in content exchange to reduce downtime and improve experience. A form of insurance, if you want.
* I wish scalable video (SVC, or adaptative video bitrate, however you want to call it) was more supported by ffmpeg & container formats. Using a single file for the various qualities would be a natural fit for Webtorrent, as it would allow HQ watchers feed LQ watchers.
* Multicast would be a great thing to have on the Internet. Maybe some future/alternative IP stack implementations could bring that back? Yggdrasil is promising, for instance. If nodes could subscribe to a given content, I guess it could be delivered across the network, splitting where it should.
That is the problem the tech community needs to solve, not storage or bandwith, ways to distribute that is really a solved issue in several ways.
The problem for creators to move off of YouTube is the discoverability for an audience.
Youtube goes to great lengths to make videos play instantly, including running huge local caches near large traffic exchange points. The audience is conditioned to see a clip start in under a second, faster for popular clips.
Anything that runs off distributed (not replicated) storage, let alone peer-to-peer links, will inevitably be slower to start. It may take 10 seconds, or 30 seconds, for a clip to begin playing.
This alone can kill such solutions for the mass audience, hungry for instant gratification.
Demanding that people accept the shortcomings of your product/service as a fact of the Universe isn't going to get you far.
>hungry for instant gratification
How many hours of video content are uploaded to the web every second? Life is too short to wait around to see if your video is worth watching in full.
A lot of people have this bias that p2p networks cannot be fast, but those days are behind us.
I think you're seeing how these systems actually work in practice, and why they operate this way. They don't quite work as you describe, and they work the way they do for the benefit of the user's playback experience more than anything else.
Most adaptive video implementations are using HLS/DASH and are serving up each video in 2-5 second chunks. (Usually closer to 2.) This is by design, and a very good thing for the vast majority of users and use-cases. You aren't beind mindlessly spoonfed chunks of the video you're trying to watch -- your playback experience is also being heavily measured, and the next chunk you receive is determined by this. If your bandwidth suddenly drops, or you've been watching a higher-quality version of the video than you can support, within a couple seconds you'll be served a slightly lower-resolution version. If it goes back up again, same thing, it'll be corrected very quickly.
If you're watching a video on your iPhone and walk from your fast wifi to the 3G outside to your garage with almost no service and back into your house again in a single minute, the video can have adapted to each of these situations quickly and fairly seamlessly.
This also solves another problem -- letting the user have near-instant playback at relatively high-quality without having to worry much about if your one-second bandwidth test wasn't as accurate as you'd hoped. The scenario you describe, where buffering for longer at the beginning to improve performance overall, is not a big concern here. If you serve a version a bit higher quality than the client can sustain, it'll be kicked down in a couple seconds. (By the way, the 2 seconds preceding every Netflix stream, when you see their logo and hear the tone that accompanies it, that's when Netflix tries to very quickly measure and determine which quality level to start you out with.)
If you have blazing-fast, highly-stable internet, that's great! And that's not most users. But in this case, most services should still be giving you far greater than 30 seconds worth of chunks, provided that the system determines this is safe/stable for your connection. If you're seeing a shorter buffer in web browsers, it's likely the cache readahead limits in Chrome/Firefox. (You can modify them!)
I can see how one might think having one big video file buffered might result in lower performance for peers, but in the ideal scenario, peers aren't being served byteranges of a single big video file that you have buffered locally with your fast internet, but rather the next chunk(s) that make sense based on the client's current playback experience.
I will admit that shoehorning adaptive video standards into p2p has been a more difficult challenge than letting WebTorrent handle the full file the way torrents have worked for years, but it's getting there, and it's the proper way to go.
Needs care with adverse risk selection, because freeloaders who want cheap video bandwidth for god knows what will crowd around.
If only IP-SSM worked on the open internet. Even some sort of best-effort routing would be a start.
The problem is not technical. It's legal.
It takes a lot of upstream bandwidth to stream even just passable quality video. Between shitty upstream bandwidth for residential Internet and swarm participants limiting their seed ratios or further limiting upstream bandwidth scaling torrent streaming is unreliable at best and impractical at worst.
* Storage costs have decreased over the years and will continue to do so, but video quality has increased during this time, as well as the number of delivery formats you need to create. For each video uploaded, you likely need to encode and store 5+ different variations.
* Bandwidth is not cheap, and while it may be lower than the cost of storage initially, for any chance of survival you'll eventually need some popular content, at which point its cost of delivery will exceed that of storage
* To prevent the video from buffering and provide an experience that is tolerable to the average user, you need to have all of the above replicated on a dozen edge nodes around the world, and serve a user the video they've requested from the one nearest to them. This is a situation where the latency differences here really can make or break the entire thing
There are numerous ways to try to optimize these things, or filter out costly/undesirable content, or prioritize only things that people are actually watching, or whatever, but few that come without consequences or that will cause bigger problems down the line.