News for hackers. Stuff that matters.
Re:Guide to Finding Lemmy Communities (Subreddits) by lolinder on Sunday June 11, 2023 @01:45PM (#36283623)
> the ActivityPub protocol is pretty inefficient when you have lots of small instances

This seems like a pretty fundamental design flaw. It means that redundancy is there—you can always find another instance if you're made homeless—but self-sovereignty was not prioritized in the design.

The kind of person who's attracted to the idea of federation is going to be disproportionately likely to want to self-host, so the system should be designed from the outset for at least tens of thousands of instances.

Re:Guide to Finding Lemmy Communities (Subreddits) by shagie on Sunday June 11, 2023 @02:13PM (#36283983)
Back in November, I was looking at what it would take to buy a hosting solution from a company that would spin 'em up.

https://masto.host/pricing/

Note the 'federation capacity' which is described further down.

If you are federating to many sites (because your users are following many users across different sites), that can get expensive in terms of time/network/cpu.

Even if you are self hosting, it is most efficient if your users tend to follow other users at a limited number of other sites so that the processing threads don't get too congested. Furthermore, if something on your instance goes viral and everyone starts boosting it and following your sites users it can significantly impact the performance.

If you are expecting to be making connections to tens of thousands of instances, the cost of an instance for that network / cpu load is going to be significantly higher than the average person can support / administer / host.

This gets to a bit of a paradox - the more you want to distribute and federate, the more expensive it is to stand up an instance and the more market forces will encourage users to go to the central hosts.

Re:Guide to Finding Lemmy Communities (Subreddits) by newaccount74 on Sunday June 11, 2023 @01:59PM (#36283813)
> The kind of person who's attracted to the idea of federation is going to be disproportionately likely to want to self-host

That's not my experience on Mastodon. Most people seem to be on shared instances. Very few people self host. It's mostly tech affine organisations who self host.

Re:Guide to Finding Lemmy Communities (Subreddits) by lolinder on Sunday June 11, 2023 @02:07PM (#36283910)
I didn't say most people would want to self-host, I said they would be disproportionately likely to want to.

To make up some numbers: if 0.001% of the general population wants to self-host, maybe 0.1% of fediverse users would if it were easier. Even with tiny percentages, there are still a lot of small instances and the protocol should have been designed for that.

Re:Guide to Finding Lemmy Communities (Subreddits) by shagie on Sunday June 11, 2023 @02:45PM (#36284342)
The protocol is fine for federation without any sort of centralization - though this causes some problems to be noted. The servers... less so. The servers have difficulty when something goes viral (and the content that it links to may also have trouble as noted in https://mastodon.social/@jwz/109411593248255294 (even though this is jwz, this link is safe))

If you have a small mastodon instance that has a member that has a post that goes viral, instead of trying to handle a handful of federation threads that your users are doing to pull content in you've now got hundreds of other sites trying to pull content from your site to theirs.

For a small site, this can completely consume all the treads.

A user on your site going viral can inadvertently DDOS the site. A widely followed user posting a link to some content can DDOS that content as every instance of mastodon wants a preview thumbnail of it now. This is known as a stampede.

https://www.networkworld.com/article/3682129/surviving-a-mas...

https://www.netscout.com/blog/mastodon-stampede

> The problem arises when a federated user with a high number of followers shares a post, or boosts (Mastodon’s term for sharing another user’s post) on their timeline. The core protocol for coordinating and sharing information among Mastodon instances is ActivityPub. When a user shares a post, it is published via ActivityPub to the Mastodon instances of all of that user’s followers. When an instance gets a new post that contains a link, it makes a request to that link to generate a preview. The process happens automatically.

> This creates the fan-in problem. A user with 100,000 followers posting a link to coolthing.example.com has effectively acted as a command-and-control for 100,000 processes to hit coolthing.example.com.

Re:Guide to Finding Lemmy Communities (Subreddits) by newaccount74 on Sunday June 11, 2023 @04:12PM (#36285126)
You are right, I misunderstood your comment.

Maybe it is still possible to fix the protocol. I think it's pretty impressive that it does work so well in practice (from a non-self-hosting users perspective)