Re:Wikipedia is swimming in money–why is it begging people t...
by jiggawatts on Monday May 31, 2021 @08:07PM
(#27349444)
Obviously I'm not being serious about hosting Wikipedia on just 3 servers. That would be a nightmare for maintenance and availability management. My point is that you could physically host their content on just one server, and nothing you have said counters that.
Who cares if Blackblaze -- a backup company -- uses hard drives? Of course they do! They're not hosting Wikipedia though. My point was by using current-generation COTS hardware, it is entirely possible to host a huge website on a single box. It is. It really is. This is not about what other people generally do. This is about what is possible!
You keep getting bogged down by the practicalities of real-world site management. For the sake of argument, if I were to engineer a serious solution for this problem, I would be looking at hardware more like what Cloudflare uses for their edge cache boxes. They did the legwork to find the sweet spot for price/performance. This is typically a single-socket server with a 48-core AMD EPYC processor and 100 Gbps Ethernet. This CPU model is much cheaper than the 64-core flagship, but has the same amount of cache. (They're made from the same 8-core chiplets, but with 2 cores fused of due to defects.)
In these servers I would use PCI-e quad port NVMe adapters, as you said. The capacity requirements aren't as high as you think either, because the content can be sharded. The 5.6 TB number is for english Wikipedia, not all of Wikipedia + Wikimedia as a whole! So those 4x 8 TB cards are more than sufficient. Heck, even 4x 2 TB is enough for a while. The trick here is to apply some sort of lightweight compression on the text. That 5.6 TB compresses down to just 100 GB! In practice, you could easily get better than 3:1 ratios even with a very fast compressor set to "light" compression.
Then I would distribute these servers around the globe, with about 3 per location. One for english, with the other two for all the other languages. Each server would have a local, read-only replica of the language-specific databases with asynchronous streaming from the master copy.
I'd distribute this to about 5 regional locations, and nominate one server per language as the "write master" where updates occur. So for each language you'd have 1x read/write and 4x read-only. These would be at different locations, so for example German wikipedia would have the read/write replica in Germany. English would be at some US east-coast location. Etc...
Then you'd need some boxes for nonproduction test, dev, etc...
All of the remote edge servers would be load balanced using a DNS-based method, with no local load balancers. The servers' 100 Gbps ports would be directly connected to the peering routers, the same way Cloudflare and the like does it. This would add up to 1.5 TBps of traffic that you really could physically serve (5 x 3 x 100 Gbps).
Overall, the entire platform could be handled by something like 20-25 servers. That's it.
Now, I tell a small lie: Wikipedia hosts only the text! WikiMedia has the pictures, which don't compress well and requie much more disk space. This is best served with technology similar to AWS S3, which can be replicated with open source software and some boxes packed with hard drives. But again, you're probably overestimating the capacity required, and underestimating how much you can pack into a server these days.
Just to reiterate: the above is predicated upon having an efficient codebase, and Wikipedia isn't currently efficient enough to make this possible. It would take a rewrite of the code to achieve this, and that would cost more than their current hardware, so it doesn't make financial sense. Most web sites are in a similar situation, which is perhaps why your perspective is skewed: Your expectations of software performance capability might be off by a few orders of magnitude!
Who cares if Blackblaze -- a backup company -- uses hard drives? Of course they do! They're not hosting Wikipedia though. My point was by using current-generation COTS hardware, it is entirely possible to host a huge website on a single box. It is. It really is. This is not about what other people generally do. This is about what is possible!
You keep getting bogged down by the practicalities of real-world site management. For the sake of argument, if I were to engineer a serious solution for this problem, I would be looking at hardware more like what Cloudflare uses for their edge cache boxes. They did the legwork to find the sweet spot for price/performance. This is typically a single-socket server with a 48-core AMD EPYC processor and 100 Gbps Ethernet. This CPU model is much cheaper than the 64-core flagship, but has the same amount of cache. (They're made from the same 8-core chiplets, but with 2 cores fused of due to defects.)
In these servers I would use PCI-e quad port NVMe adapters, as you said. The capacity requirements aren't as high as you think either, because the content can be sharded. The 5.6 TB number is for english Wikipedia, not all of Wikipedia + Wikimedia as a whole! So those 4x 8 TB cards are more than sufficient. Heck, even 4x 2 TB is enough for a while. The trick here is to apply some sort of lightweight compression on the text. That 5.6 TB compresses down to just 100 GB! In practice, you could easily get better than 3:1 ratios even with a very fast compressor set to "light" compression.
Then I would distribute these servers around the globe, with about 3 per location. One for english, with the other two for all the other languages. Each server would have a local, read-only replica of the language-specific databases with asynchronous streaming from the master copy.
I'd distribute this to about 5 regional locations, and nominate one server per language as the "write master" where updates occur. So for each language you'd have 1x read/write and 4x read-only. These would be at different locations, so for example German wikipedia would have the read/write replica in Germany. English would be at some US east-coast location. Etc...
Then you'd need some boxes for nonproduction test, dev, etc...
All of the remote edge servers would be load balanced using a DNS-based method, with no local load balancers. The servers' 100 Gbps ports would be directly connected to the peering routers, the same way Cloudflare and the like does it. This would add up to 1.5 TBps of traffic that you really could physically serve (5 x 3 x 100 Gbps).
Overall, the entire platform could be handled by something like 20-25 servers. That's it.
Now, I tell a small lie: Wikipedia hosts only the text! WikiMedia has the pictures, which don't compress well and requie much more disk space. This is best served with technology similar to AWS S3, which can be replicated with open source software and some boxes packed with hard drives. But again, you're probably overestimating the capacity required, and underestimating how much you can pack into a server these days.
Just to reiterate: the above is predicated upon having an efficient codebase, and Wikipedia isn't currently efficient enough to make this possible. It would take a rewrite of the code to achieve this, and that would cost more than their current hardware, so it doesn't make financial sense. Most web sites are in a similar situation, which is perhaps why your perspective is skewed: Your expectations of software performance capability might be off by a few orders of magnitude!