what, why? The load is hardly that high.
An actual takeover will almost certainly be a virtual DDoS on Mastodon.
For reference my work laptop (8 logical cores, so 4+HT? 32GB RAM) can handle 100k rows/second sustained inserts into postgres 14 with some batch jobs I'm working on. You can buffer http requests into batches and easily handle way more than 10k/s on a server while still providing synchronous semantics and reasonable latency to the client (e.g. flush batches every 100 ms).
I doubt Mastodon is designed for that kind of scalability, but most techies could probably afford to run Twitter as a hobby if they knew what they're doing and they weren't trying to do all the analytics stuff to monetize it/just wanted to provide the service.
You have no idea what are you talking about.
You can easily batch writes into neat transactions: make a queue, have your POST handler write (row, callback) onto the queue and await the callback. Have the queue reader grab a chunk, push it to the db in batch, and execute all of the callbacks on commit. The callbacks return a 200 to the client, or 500 if the commit failed. This can all happen fast enough to be done in "real time" (however fast you want your queue worker to flush batches).
You can do all of this in a couple dozen lines of code with something like Scala/ZIO.
Computationally, it is totally doable. The biggest constraint is the cost of storage.
When I said that you can't batch, I meant that each tweet will be at least one transaction. Async write with batching like you suggested will have a horrible user-experience, also your client often is a mobile device or browser - how do you deliver a callback from server there?
Nah, how many inserts your laptop can make and how many tweets created in a unit of time are two irrelevant metrics.
You don't. The callback is on the server before it ever responds to the request.
Here, I slapped this together to demonstrate the technique[0]
Your HTTP route handler becomes
override def create(body: String): Task[Long] = {
for
rspP <- zio.Promise.make[Nothing,Long]
_ <- createQueue.offer(InsertRequest(body, rspP))
rsp <- rspP.await
yield (rsp)
}
i.e. make a promise, put your work on a queue, and have the HTTP response be the result of the promise. Then you have a background worker: _ <- ZStream.fromQueue(createQueue)
.groupedWithin(8192, 10.milliseconds)
.run(ZSink.foreach(repo.createChunk)).forkDaemon
Which processes up to 8k requests at a time, waiting up to 10 ms for a batch.The worker does a bulk db insert, and completes the promises with the generated ids.
Similar techniques should work on read batching, but I haven't tried that. You can also speed that up some more with the COPY protocol, but IIRC you need to be more careful about escaping/SQL injection.
On my 6 year old mid-range desktop (this CPU[1] and this disk[2]) this program can process ~30k `create`s per second. For about $1500, I could buy a new computer with a Ryzen 9 7950 with 4x the core count/8x the thread count and 2x the single-threaded performance, so around ~10x more processing power, 128 GB of RAM, and a Samsung 980 Pro SSD, which can do 1M Write IOPS (25x more than my SSD) or 5GB/s sequential writes (10x more). So a $1500 computer with a single disk should be able to do around 300k/s. PCIe gen 5 is now coming out, which will allow for another doubling of disk performance.
128GB of RAM means you can keep at least 100M rows worth of index in memory. It's not that expensive (under $10k) to build a server with 1TB of RAM.
Totally feasible for a hobbyist to do without tons of tricky optimization (the code I posted is purely functional Scala!); people spend $20k on a jetski or $80k on a truck. Like I said, the most expensive part is going to be the storage, but you could do something like only store the most recent 1000 tweets per person, and charge $10 to bump that up to the most recent 10 million tweets or something. You'd come out at a substantial profit with that model if you got a few thousand takers. Similarly you could charge to let someone follow more than a few thousand people so you could pay for a read replica or two.
[0] https://github.com/ndriscoll/twit/commit/19b245677b978b42a6f...
[1] https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i5-6600K...
[2] https://www.disctech.com/SanDisk-SDSSDHP-256G-256GB-SATA-SSD
> By volunteers adding more instances(that they can close anytime)?
"That they can close anytime", just like twitter in this example. Partly, yes. But that sort of total exodus would mean a lot of additional people contributing ideas and code to the Fediverse, not just servers, but by making it easier to run your own instance. Who's to say it couldn't be run just like an email client with the right ideas and effort? It's such an extreme example that I'm not even sure it's useful to discuss.
What traffic is it prepared for? It would be interesting for you to provide the numbers and the evidence which backs this up.
As for real life scenarios...there are upwards of a million people using it right now. I've made friends, networked professionally and found several homes there. I am literally a real life scenario and so are the people behind most of the posts there.
And today you are right about "anyone can start their own instance", but it's a darn sight easier than running your own twitter.com, and it'll get easier every year.
Twitter pays people money to keep their service running, so there's that incentive.
> As for real life scenarios...there are upwards of a million people using it right now. I've made friends, networked professionally and found several homes there. I am literally a real life scenario and so are the people behind most of the posts there.
Twitter is, say, 300m MAUs. That would mean the volunteer Mastodon infra would have to increase 300x (assuming scaling is linear, and the Mastodon community hits Mastodon as hard as Twitter users hit Twitter) to cope with similar traffic numbers.
For servers that run Mastodon.
> But that sort of total exodus would mean a lot of additional people contributing ideas and code to the Fediverse, not just servers, but by making it easier to run your own instance.
Most users aren’t interested in contributing anything to the platform. Social media platforms popularity lays in simplicity. No one wants to run anything, just use service without any hassle.
> there are upwards of a million people using it right now.
Compare that to 200m+ users of Twitter sending 500m+ messages daily. I bet Mastdon can handle this without a sweat.
Running a server that can cope with thousands of users would probably cost just a few dollars a month. Donations would be more than sufficient.
It is not early days anymore and no non-technical user is interested in hosting their own servers for chatting with another person. They don't care about decentralization as even if they tried they will recentralize to the main Mastodon instance.
Also, these users don't even know which instance to go to, since there is little to no-one to talk on there. If there are 'hundreds of thousands' of users then that means they have just recentralized on Mastodon.social, the "main" instance, defeating the point of it all.
> PS. Please do not say anything about “anyone can start his own instance”. No, average Twitter/Facebook consumer can’t start his own instance.
This is why Mastodon has failed in the first place after almost 6 years with this system.
There are plenty of people using it. It has not “failed”.
Yes, I said: 'Little to no users'. After looking at it for a couple of years, it is not the typical twitter user that is self hosting their own Mastodon instance and just the same tech-folks that are doing that (unreliably) and sitting on Mastodon. The level of social interaction on Mastodon is so low and limited, that they still use their Twitter accounts more than their Mastodon accounts.
So yes, it is not early days anymore and we have given it enough time and it has already failed.
Having 90% of registered accounts inactive with only 10% of them actively using the platform isn't exactly 'alive', 'growing' and 'healthy' especially when they occasionally run back to Twitter since they know little social engagement goes on Mastodon. 10 is closer to 0, than 90; Hence "Little to no one".
But we both know it is not just that. Not only they can't help using Twitter more, they won't move to Mastodon for the exact same reasons as I said and Twitter's network effect, hence why little to no-one is using Mastodon. The same tech-folks like (Mastodon.technology) are the ones 'self-hosting' these instances and not the regular users, since they don't care enough to even use it.
Not even the one operating Mastodon.technology could handle it. Might as well recentralize back to Mastodon.social just to save itself from the very low levels of social interaction since Mastodon has already repeated the same problems as GNU Social once again.
That's not at all true. Account activity follows the Pareto principle. It's not at all unusual for any online service to have a large number of inactive users. Perhaps it's different for Twitter, but considering I've probably signed up for it three times and use it approximately never, I'm skeptical.
https://www.google.com/search?q=rvz+mastodon+failure+site:ht...
I wouldn't engage.
Repeating what? The truth?
Right, Please don't engage because everyone here knows it is true.
Resources donated by an organization in the form of a server linked into a larger network, a committee that vetted new server applications to the network, volunteer administrators for the network and the individual servers, coordinated regional and global upgrades. And as network users increased, reforming under a hub and spoke models to improve scale and capacity.
And when a single IRC server went away after some time operating for its various reasons, the network kept going.
Could the average IRC user start/host their own instance *and* link it to the larger network? No. But they didn't need to.
It is ok if it never gets adopted as mainstream communication. But for the target audience (like opensource support communities being the target audience of Libera), it works quite well.
In my opinion, the main reason why it was never adopted as a mainstream platform, is because it was never picked up by a big corp that saw a way to earn money off of it.
Insert any modern chat app and random user can drop a screenshot there trivally, do some basic text formatting, and paste code in nicely colored blob of text, all from comfort of single login in a browser. And now even jump to voice chat or video conference directly from it.
They can also look at what they missed in channel by just scrolling up. Sending files just worked. You just got notifications without extra fuckery. Hell, you get digest with conversations you miss.
That's why modern closed down chat ecosystems won. Coz it's easy for "normies" to use.
Let's take other example, XMPP. It had all that but in network of XEPs that some clients/servers implemented, some not, some just not very well, and some being fucky to config. It wasn't just "login here and get every feature" like it is in modern chat clients.
It had it's one shining moment where both facebook and google offered XMPP way into their garden so you could have connectivity across federation and just have a single contact list but that didn't matter to non-nerds that just used facebook/google talk so once they decided to close their gardens any benefit of xmpp faded away.
I haven't forgotten about the Freenode hostile takeover.
I disagree with that characterization.
> and survived
Survival is not the issue. Mastodon will survive. Tumblr survives. Even MySpace survives. But major disruptions tend to lose users.
(And yes it's true that the potential Twitter acquisition is a potential major disruption. But it's not going to be shut down after a $44 billion investment.)
As a user, I only had to point my client from Freenode to Libera (exactly one line change in my client config), run /msg nickserv register to register myself, run /msg chanserv register to register the channels I op-ed, and it was all done.
Total time spent was less than 30 minutes. The next few days, others did the same and the community started trickling in to the channels in the new servers. Seems seamless enough to me. I doubt such an easy migration is possible if Twitter disappears suddenly.
It's probably more about how much the new owners will want to drop into it and how long before it moves to x.com (?) and becomes an everything app
Why did you feel the need to mention this 100% obvious fact?
Of course I meant that the new owners wouldn't shut down something they just spent $44 billion on, thereby throwing their investment in the trash, not that Twitter would magically get a $44 billion operating cash infusion.
It was not 100% obvious what you were implying, obviously
Okthanksbye.
> It was not 100% obvious what you were implying, obviously
"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize." https://news.ycombinator.com/newsguidelines.html
Be kind. Don't be snarky.
---
I see you've edited it now to be more clear
The same people who pay for everything right now will pay for it: us. Some instances will have patreon, others will be voluntary donation, others will use some craptocurrency, others will have contractual subscriptions, some will have ads... And whatever models are best will win out. Quit with the FUD. Just sit, back, relax and watch it happen.
The official instance finding site seems to be good about spreading the load out every time Twitter burps. You have to meet certain reliability requirements to even be listed.
The whole "millions of users" fallacy is the result of people not being able to grasp what federation is about. The network can easily accommodate millions of users. Individual instances don't need to be able to.
Of course not. Mastodon instances can be capped in the first place, and anybody with a rudimentary server management knowledge can start their own instances on a cheap server. Mastodon has current hundreds of instances, let's not pretend it can't go to thousands if the user base increases.
This isn't a law of the universe, it's just software people haven't written yet. Installing new client-side apps was hard, until it wasn't. "Anyone can start their own instance" will be easy once someone writes the software to make it so. (Presumably a cloud provider like AWS, since that's who stands to profit from lots of people wanting to run server-side apps)
Who said anything about the Fediverse having to be free ?
There is absolutely no doubt that should Twitter die, if no single actor can emerge quickly enough, for-profit actors will emerge and they will have all good reasons to be compatible with something that already exists. There will be mega large instances paid by siphoning data and with ads, there will be large instances paid by users/funds/donations, there will be small, community instances. Maybe HN will have its own instance; how much do you pay for HN today ?
For a decentralized social network to be viable/sustainable (especially on the scale of something like Twitter), it has to be truly P2P, not federated on volunteer-run servers paid for through donations. That volunteer-run federated model is really only sustainable for smaller niche communities, not a global social network.
As of right now, the closest framework I can think of to handle something like this is a social network built on OrbitDB: https://github.com/orbitdb
Now first storage space taken will be ridiculous, and then you will be hosting content you don't want to host because it could be straight up illegal in your country.
Look at any crypto-blockchain and count how many full-nodes vs thin-nodes are running in networks.
Then there is the whole tragedy of commons: average seed ratios and streaming video players for torrent content shows that are not willing to contribute to the greater good.
There is an option of dividing it between "publisher" (source) and "cachers" (caching whatever is recent and popular) but that only helps a little bit.
> Then there is the whole tragedy of commons: average seed ratios and streaming video players for torrent content shows that are not willing to contribute to the greater good.
Over 2 decades of pirating that has only been a problem for obscure and old stuff.
I don't see it being a problem in social network sense; you publish your stuff, if there is nobody wanting to look at it, seeds 1, leeches 0. Moment there is interest it propagates. It is of course problem if you want to see something archival that author is no longer in the network tho.
The Planetary app handles this much better, but it’s only on iOS/iPadOS ( therefore also able to run on M1 MacBooks) unfortunately.
It's a common problem for old stuff like you said, very new stuff with specific formats. It's an issue for new stuff because it often doesn't have enough starting seeds, and people leaving as soon as they're done watching/downloading doesn't help the situation. Before I've discovered usenet this was a very common issue for me.
Then there is specific file formats like chunky movie rips with Dolby Vision + Dolby Atmos + Dolby Yet Another Audio Format + all of that again in 5 languages. People will often choose a different release to download, and that leads to fewer seeds.
I know a group of people on a popular torrent tracker that just downloading everything and seed forever. They do entirely for free. Those petabytes will probably rot once they...die.
The idea that social media costs more to operate than people would be willing to pay is false. It's propaganda from the people who profit from keeping you trapped in their closed networks to monetize your attention.
Server written in reasonably fast software on $10 (well, probably some extra bux in storage if community is image-heavy) VPS can reasonably serve tens of thousands of users in typical forum/blog/hn-esque format. Mastodon isn't but I don't doubt someone would rewrite it in something faster if there was a need for it.
So I do not see any advantage in federated system. It’s cool as technology and all, but completely unprepared for huge traffic or real life scenarios.
PS. Please do not say anything about “anyone can start his own instance”. No, average Twitter/Facebook consumer can’t start his own instance.