You colocate, like every other distro has done since the 90s.

And throw in some rsync mirrors for good measure.

(piping up from another thread, but)

> I'm very interested if you happen to know a cheaper alternative!

and

> Optimally you'd want a distributed CDN

https://www.storj.io/ is this thing!

Using the pricing from your other comment and the numbers from the parent, if I estimate correctly, Storj comes in at about $8k/month? That's cheaper than AWS (considering there are no CDN costs), roughly on the same level as Cloudflare and Backblaze, but still more than the entirety of NixOS Foundation's (monthly) incoming funds in 2022.

According to https://discourse.nixos.org/t/nixos-foundations-financial-su..., they have 450TB and 29.5TB of S3 egress/month. By my math, assuming the Nix Foundation goes in through our front door, talks to no sales people, and pays list price, that's $2k/month.

You would replace the Fastly CDN with Storj, correct? Your link mentions 1500TB of Fastly traffic, which would cost $12k with Storj. Not sure if this is related to parent's 3TB/day number, but I went with that when I estimated the $8k.

I think the best cloud solution might be something like Storj + Fastly or Cloudflare end-to-end. But I'm really curious how much could be chopped off if you tried to solve this on-prem.

Storj + Fastly is genuinely a pretty good combo. There are certainly some things Fastly does that we don't do (besides being more specifically a CDN, they also have compute@edge, etc). We have a good relationship with Fastly (e.g. https://docs.fastly.com/en/guides/log-streaming-storj-dcs)

But yeah, I don't know how the Nix Foundation is getting their Fastly traffic paid for, maybe that's in/out of scope. Certainly they should keep Fastly if it makes the overall thing cheaper. Having a distributed origin (us) to dedicated edge cache (Fastly) makes for a strong distribution story.

Actually, for someone building an OS, having to face very practical challenges in the server space would likely be extremely beneficial. Dogfood, people, dogfood.

> Plus, even if you deal with the storage itself, how do you deal with serving the traffic globally?

As a simple example 1 of the cheapest hetzner baremetal nodes gives you 500gb nvme or 2tb HDD storage with an unlimited gigabit Connection in Europe. This sums up to i believe 10TB of data per day.

Given the provided numbers, a single 50€ instance could handle the load, even if it's located in Europe. CDNs make sense for systems which are sensitive to high latency, or you're pushing multiple TB per hour, which this just isn't.

I wonder how unlimited such cheap unlimited connections really are. In the simplest case insufficient capacity in the provider network might become a bottleneck. They might notice that you are the noisiest one in the whole neighboorhood and either ask more money or throttle you. Offers that are too good to be true typically don't last very long.

At 1 EUR/TB, even the instance types that pay for traffic (e.g. the 10GBit dedicated servers) hardly break the bank.

However Hetzner does not have the best peerings, so bandwidth to some locations might be worse than you'd like.

Thats true, and I didn't actually want to suggest using the cheapest server available as a singular node either.

I just wanted to put the given numbers into the scale of the cheapest baremetal servers available on the market, to illustrate that this is not an amount of data you'd have to consider for scaling. Realistically speaking, you'd want to have more then 2 nodes so you don't have to worry about rebooting/updates etc

I have quite a few servers with hetzner. I don't know what the exact criteria are but once you hit around 250TB/month per server with multiple servers for (likely) multiple months in a row then they will tell you to chill out or they will terminate. I've gotten it and I've seen a few others on lowendtalk mention it too.

I've been using hetzner for a couple years and have always used them for my high bandwidth services but I have only gotten that notice once despite being a very heavy user, and around the same time everyone else did around a year ago. Maybe they're gonna start to care? With all the price hikes and all. I ended up just getting more servers to level it all out.

I’ve run some fairly network-heavy services (900mbps+ on a 1gbps connection, 24/7) at both leaseweb and hivelocity; they never complained about the traffic

It is astonishing as to how marketers from the three big clouds have successfully convinced people that there is no middle ground between operating your data center and using one of their cloud services.

Here's what you do:

1. Buy two servers equipped with a large amount of storage. There are many providers[1][2] that offer this. (You could technically do it with one, but two allows you to have high availability and prevent data loss in case of accidents, imagine `rm -rf *`.)

2. Use Minio to set up erasure-coded storage on one of the servers, and replicate objects[3] to the other server.

4. Use Cloudflare (because of their open source sponsorship program[4]) to serve the objects from the primary server, and when performing maintenance tasks, fall back to the secondary.

You could also serve objects directly out of the primary/secondary, but using Cloudflare takes care of low-effort DDoS attacks by hiding the origin, and provides some caching which helps prevent hitting the egress limits of the server provider.

[1] https://www.netcup.eu/vserver/vstorage.php

[2] https://www.hetzner.com/dedicated-rootserver

[3] https://blog.min.io/active-active-replication/

[4] https://blog.cloudflare.com/cloudflare-new-oss-sponsorships-...

Yes, perfectly doable, but the expertise and time needed to do it (without fatal mistakes) and keep it running 365 days a year is non-negligible. Doing this by unpaid volunteers is more than challenging.

This is, in my opinion, a falacy perpetuated by the marketing departments of "big cloud". Don't forget that pre-2006 there was no S3, and we managed just fine. They are not challenging skills to learn, the zeitgeist has just convinced itself they are not worth learning

It's condescending to think that someone can't reasonably choose to pay for a service they are capable of doing.

Cooking isn't hard either but I still go to restaurants, and don't go around laughing at patrons telling them that they never bothered to learn how to cook if they are there.

Eating in the restaurant once in a while is fine. Lage range of choices delivered quickly. But is it sustainable? Price higher, usually less healthy and not tailored to special needs.

In any case one does it professional IT done right is costly but cheaper than e.g. IT without offsite backup in case of a disaster.

Do you eat every meal at a restaurant though? That would not be sustainable. To roll with your analogy it would be like the hospitality sector persuading the majority to ditch home meals entirely

The problem was your assumption which is condescending. I gave you an analogy to make it clearer. Taking the analogy apart doesn't change the point.

It's not condescending at all. You are saying they could reasonably pay for it, but they quite obviously cannot (hence the existence of this whole post). What I'm principally challenging is the assumption that this required effort is substantial. I'm making the point that it is not, certainly (in my view) compared to the effort to run an entire operating system project and all of the ancillaries that come with that. This is an area they have chosen to pay for that (again, in my view) they don't need to. They have gone to great lengths to architect a system that allows deep pinning of very specific versions, it's not a stretch to think about the architecture required to deliver that. Instead they have stopped that line of thinking at [insert S3 here] rather than carry on solving. A direct consequence of that now is a $9k/m hole they need to fill in their budget. I'm being critical, sure, but not condescending. Overall, it frustrates me seeing opinions that operating infrastructure is difficult and therefore must automatically be avoided. In this particular example avoiding operating said infrastructure potentially is putting the project in the red and at risk, but nowhere on the NixOS call for help does it suggest they have considered operating this themselves. And there in is the crux of my point. The zeitgeist has turned against operating infrastructure, and yet is frequently caught out by the cost of this decision.

> They are not challenging skills to learn, the zeitgeist has just convinced itself they are not worth learning

Next time you see someone paying for something you think it's easy to learn, that the "zeitgeist told them not to learn", go up to them and say what I quoted and you'll see real fast if it's condescending or not.

This is a silly thread.

The difference in situations here is literally that on the one hand large corporations have spent money to market the idea that their services are essential because the alternative is incredibly difficult. On the other hand, McDonalds, KFC, Burger King, "Big Restaurant" etc have not spent large quantities of money trying to persuade the populace that cooking at home is incredibly difficult. It's possible to sell someone a convenience without also telling that person that it's the only real option and this is what the person you're replying to is highlighting.

It's not condescending, it's the reality of the situation. Yes in the examples you've described it would be condescending but in the situation we're actually talking about there are actually large numbers of people who genuinely believe (as a result of successful marketing campaigns) that the equivalent to cooking your own meals at home is incredibly dangerous, difficult, expensive and error prone.

> On the other hand, McDonalds, KFC, Burger King, "Big Restaurant" etc have not spent large quantities of money trying to persuade the populace that cooking at home is incredibly difficult

They haven't because it's a ridiculous proposition. The same way cloud providers would be laughed out of the room if they tried to say the same thing and they didn't actually have a better product with trade-offs that are genuinely better for some use cases. The fact that you believe that everyone using cloud products is either too lazy to learn or has been fooled by marketing because they can't think for themselves, unlike you who see the light, is the condescending part. Mind you, on the face of overwhelming data showing most companies have, or are moving large parts of their infrastructure to the cloud, so everyone else are dummies either too lazy or too stupid to see what you can see.

> The fact that you believe that everyone using cloud products is either too lazy to learn or has been fooled by marketing because they can't think for themselves,

I never said anyone was too lazy, just misinformed about the benefits, and it's a more far fetched statement to claim that marketing doesn't work than it is to state that it does work.

As evidenced by some people in this thread claiming that not using S3 would be basically impossible (despite a large number of popular distributions literally doing exactly that) there are clearly plenty of people who have fallen for the marketing.

Moreover, you're the one implying that falling for marketing is something only people who can't think for themselves do, that's also not what I'm saying. I think everyone is susceptible to marketing, and in this case, clearly a lot of intelligent people who are otherwise capable of thinking for themselves have fallen for this blindspot.

I think you are the one being condescending here.

The original post did not come off as condescending to me, but instead just highlighted how BigCorp cloud marketing departments have an incentive to sell developers on “infrastructure is tough, just pay us”. (I say this as a consumer of AWS services and fan of their products).

“Assume good faith”[1]

[1]https://news.ycombinator.com/newsguidelines.html

Reading back over that, I'm not seeing the condescending bit?

Who managed just fine? IT-based services very a minor fraction of what they are today. And IT departments in some random company were much bigger.

> And IT departments in some random company were much bigger.

Not sure about that. There appear to be just as many people in IT, it's just that they're not that technical anymore, mostly producing policy documents and pushing tickets around. The actual work is outsourced to the cloud, SaaS products and outside consultants (for duct taping the mishmash of SaaS products together). But that may be just my lacking understanding of my strange little part of the globe.

Not for the people behind NixOS. If it is challenging then that would cause me to re-calibrate my opinion of the project as a whole.

We do this at work, albeit much larger scale than this and I'm the only one who spends any time on upgrades, replacing failed disks and so on. Honestly, apart from upgrades every 3-ish months I don't spend any time at all on this, it's all chugging along without intervention.

Things don't fail as much as they used to, at least not from using something mature as minio and of course hardware can fail as they do specifically spinning rust, which is non trivial to replace if you have hot-swap capable servers in the first place.

I don't possess any decade worth expertise in either minio or running this kind of setup so personally I believe you are exaggerating that this is even worth calling challenge at all.

But that's just my take :)

So I’m sure you’d be more than happy to volunteer your services to NixOS and be responsible for the setup, including when there are issues (which will inevitably happen regardless of tools and technologies used), right?

It’s so easy, after all.

I'd volunteer to do it, if they paid for hardware, colo and transportation for initial install. After things settle down, it takes no more than half an hour or so every month.

Pretending to know how much work something is when it's not your area of expertise is a bit silly.

Actually, yes, it is easy, after all :)

The money that are paid as the exorbitant AWS egress fees could be used to pay a competent SRE to handle that. I don't think that running the package mirror would require a full-time dedicated paid position. Initial setup, maybe some heavy restructuring operations as storage requirements grow, maybe some firefighting, but not the daily operation.

I mean, you make S3 look easy compared to that

That's the business model.

If `minio server /mnt/sd[abcd]1` and the occasional `apt-get upgrade` and changeover of DNS records is so much work, maybe you should just spend 100x for S3.

Seriously, there is a large amount of comments of people who seem terrified of this or throw out "but but updates and failing hardware". We run multi PB storage on prem at work and it's just fine. Not some massive team running it.

You do all of this and you still don't have nearly the availability or durability of data you get from S3, nor people on call to fix it once anything happens to it.

It's like comparing leasing a Mercedes-Benz with full service and all inclusive insurance to building your own bicycle. Some people don't need the car, sure, but it's not the same thing.

But they're complaining about the price of leasing the Mercedes, so they need to lower their expectations if they want a cheaper solution.

Although having to spend some time to attain good enough availability/durability is not a big deal when the rush to launch was over long ago. Unlike in your example, you can turn the bicycle into a car.

The usecase is a cache. http://cache.nixos.org/

All it needs to do is to get you the data faster than the compilation itself - most of the time - to be useful.

I also wonder at the original thinking, given the use case. Cache usually has high ratio of download to upload and doesn't need high reliability. S3 doesn't excel at the costs of fetching the data from it, at all.

It also “caches” the source code downloads, many of which are no longer available from the URL listed in the repository, if anywhere at all.

It only seems like that to people who don't know any better.

Do you ACTUALLY think you can communicate with people at Amazon who truly know what's going on? If so, why do we see countless stories about people who have problems with Amazon, Google, et cetera, who can't communicate with them? Do you know of some special, magic way to get the skilled people to talk with you that the rest of us don't?

You act like being able to call Amazon, even if you could get someone on the line who knows something, is somehow better than having even just one good systems administrator. All of your ridiculous assumptions go right out the window when you have a sysadmin build, test and deploy your own hardware that you (via your sysadmin) have complete control over.

Security? You can't secure Amazon. You literally can't. If you wanted to, you'd have to buy the company, then you'd have to evaluate your whole S3 staff. If you hire a systems administrator, you have to vet your systems administrator and anyone else who has admin access, not a collection of tens of thousands of employees across the planet, any of whom could be stupid enough to visit the wrong web site using the wrong browser on the same computer as their keys.

Availability? My side projects have better uptime than Amazon.

Durability? Two geographically disparate locations and LTO, plus many mirrors, make things plenty durable.

When you're not a systems administrator, you can be forgiven for not understanding the big picture, but not for acting like you do.

Yes it is very simple, you pay for support. Maybe I don't know any better, working professionally on top of AWS for almost 10 years and growing usage from a couple of thousands a month to half a million a month (while remaining profitable the whole time). Maybe, just maybe, people disagree with you even if they are informed.

And for the point in question, you don't need to talk to anyone, the point is there's thousands of datacenter, networking, app-level layers of staff all working to make service stable for you without you even knowing. I don't need to talk to anyone if an S3 server, rack, or even full DC goes down.

Honestly, we are an open source project, not a company. If we have the money, of course, we would prefer to spend it there and do better and more with it.

If this is threatening in a serious countenance, our growth and ability to conduct our project. I'm not sure if we need all the 9s of durability of S3, given that people already lost all their buckets on S3.

We do not have the money to store 5x or 6x the contents of the cache.

So we much prefer to have control, and we have enough skills to run all that shit easily, the problem is that the skilled people are in rare availability and are usually working on harder problems than running that. So ultimately, this is a balance problem.

Disclaimer: NixOS developer, 23.05 Release Manager.

In the olden days you’d find someone to donate you a server and have a few volunteers around the world mirror it.

It still happens. You need storage a TB or two with unlimited (or at least very high) traffic? That's not an issue. Even I could help being a mirror from home. But I wouldn't be surprised if there would be some free tiers already from cloud provider.

Half PB is a bit more than that, and I guess it is expected to grow... in 2023 it is not realistic to throw such storage around for free (+mirrors). It will be realistic in a few years considering that the storage prices have been in free fall for the past 30 years, but that time hasn't came yet.

(As a maybe interesting comparison, Debian's mirror is around half TB: https://people.debian.org/~koster/www/mirror/size )

The falling cost of storage is at least partly the reason hosting a few thousand packages takes up hundreds of TB. "Storage is practically free" from the perspective of each of those projects. By the time .5 PB is a trivial cost, they'll need 50 PB.

Yeah, that too.

Yeah NixOS has done some technical choices to require tons of data downloads, even if there is little modification of the content of the packages. Regularly my computer has 10 Gb worth of downloads because some high level package got upgraded and even if it's dynamically linked, the lower level packages still get re-downloaded from cache.nixos.org. It doesn't matter because storage is free right?

You really don't need al that mess, just a bunch of mirrors around the world.

Or a couple SX servers from Hetzner

https://www.hetzner.com/dedicated-rootserver/matrix-sx

(You would need 4 SX294s for half a petabyte with 2x replication)

And put ceph on them which gives you an S3-compatible API:

https://docs.ceph.com/en/quincy/radosgw/index.html

It's not trivial to set up but once you do it's much cheaper than AWS. It's even cheaper if you can find the machines in the server auction. There are some 150 TB machines on there right now for 174 euro/month.

While I totally agree you can easily handle this with Ceph, I would never do it with 2x replication on spinning rust. You can maybe do 2x replication with SSDs if you really know what you're doing and you have data replicated on multiple Ceph clusters.

Also, your math does not quite add up if you want to store 425TiB. You'd be filling up the Ceph cluster to 95% which would be impossible (you'll start losing OSDs way before that).

If this was me, I would definitely do this on Ceph, but I would opt for a more traditional erasure coding setup of 8+3. For that, you'd need 12x SX134s as a minimum starting point.

I suppose you could start with an EC profile of 2+2 or 4+2...

I do have a serious question on Hetzner:

Do they offer any private networking options? That page only lists a single 1 Gbit connection. I don't think that would work well with Ceph. It's doable, sure, but I would want to support such a cluster, not with 160TB raw per box.

> Do they offer any private networking options?

Doesn't seem like it for their dedicated servers, not 100% sure though.

Their docs indicate that upgrading the 1Gbit ethernet link to 10GBe is an available option:

https://docs.hetzner.com/robot/dedicated-server/network/10g-...

You can upgrade them to 10gb NICs.

Internal networking in simple case provided via VLANs on that 1gbit interface via their vSwitch.

Extending with additional NICs should be possible from what I've read but not used myself.

On the technical side, garage (https://garagehq.deuxfleurs.fr/) does multi master replication by default, so is probably better for this use case. Still with S3 API.

I have a few projects running off of a self-hosted garage instance, it’s very nice software.

> Even if I had the hardware, as a pure software person, I have no clue how I'd even start setting up a physical location to serve data on that scale.

3TB/day is about a third of what you can serve out of any run of the mill PC with a gigabit ethernet connection. There are 22TB drives and individual servers with 24 drive bays, which exceeds 500TB. So you could do this with one machine connected to a gigabit fiber connection, and the machine would cost approximately what they're paying monthly (almost all of that for the drives).

In practice a 24-bay chassis is silly and you might use a separate storage array etc., but that's just details.

> Plus, even if you deal with the storage itself, how do you deal with serving the traffic globally?

At 3TB/day you're not Netflix. Just serve it directly from your own server.

https://www.supermicro.com/en/products/system/4U/6048/SSG-60... - 4U 72 drives

It's not silly because it doesn't exist. It's silly because it weighs a hundred pounds and makes design trade offs to achieve a level of density you probably don't even care about if you only need one of them. The people who want that thing are the people who are going to fill a rack with them or more instead of several times that many racks with something else.

In practice if you need ~24 drives you get a couple of 2U array containers that each hold 12 drives and then buy a 2U server or two to plug them into and don't care at all that you used 8U for 24 drives instead of 4U for 72 because your 42U rack still has 34U to spare.

Well, having a $9k per month budget you could hire an entire rack in a premium data center location (let's say Telehouse London-Docklands that hosts London internet exchange - You can connect to it in a very cost effective way). This would still leave 80%~90% of the budget (depending on power use) for hardware/software etc.

But a full rack is a huge overkill. How about a quarter rack for few hundred $ per month?

What about connectivity? Well, as mentioned Telehouse Docklands hosts LINX so you can get an unlimited transfer gigabit right into the Internet backbone for a couple hundred $ (mind I dealt with them few years ago, but I doubt prices changed that much - probably they're lower if anything). Also there is a cluster of data centers in docklands so they compete on price heavily as well as resellers that often have good deals.

This is all just location and connectivity. How about hardware? Let's say they wanted proper support with their hardware not diy so they were looking for a "solution provider". Send a request for quote to 10 of them including fujitsu, ibm, hp etc. Then tell them you're looking for a "charitable organisation discount". Unfortunately we're missing the most important metric to price this "solution". IOPS and if its mostly small or large files (random or sequential Access), but they host most of it in "infrequent access" s3 so let's assume it's not a lot, just to find the lower pricing boundary.

So let's say a pair of 2u servers with two hba cards each and 4 12 disc jbods with 22tb Sas drives a total of a petabyte in raw storage or half a PB mirrored (not that they would run it mirrored). A system like this from a big name (it's rather low performance, but we're after cheap right, their s3 is likely very similar) might cost $80k. Over 5 years of lease that might be ~$3k per month plus their hosting and connectivity it's about half of their $9k.... Then there is software, but that's a separate thing. Also one could DIY such system for half the price, one could add SSDs etc. This is just an example. To design this properly one would need usage data from their s3 and few days of work to put it nicely together.

> mentioned Telehouse Docklands hosts LINX so you can get an unlimited transfer gigabit right into the Internet backbone

First, look at the LINX network map. They are not just in Telehouse Docklands.

Second, with all due respect, do you know anything about how the internet works ? "internet backbone" huh ?

LINX is a peering exchange. A connection to LINX gets you nothing. Sure you can peer with the route servers. But you won't find any Tier 1 routes on the route servers, and the Tier 1s will also refuse to peer with you because you're too small or you don't meet their ultra-restrictive peering policies (for example, good luck getting BT to peer with you !).

Most people are better off getting a good connection to either a Tier 1 or a high-quality Tier 2 network and leaving it at that.

> Also there is a cluster of data centers in docklands so they compete on price heavily

London Docklands is the most expensive place in the UK to colocate, and probably in the Top 5 most expensive globally. Its prime real estate, all of the big-name colo facilities know that and will charge you accordingly, ESPECIALLY Telehouse !

And trust me, you don't want to go anywhere near a "cheap" Docklands facility, they are cheap for a reason.

The days of £300 a month for a full rack in Docklands are long gone my friend.

There are many other perfectly good places both in the Greater London area and elsewhere in the UK and Europe where you will get more bang for your buck.

I’d be surprised if any of the top five most expensive colos weren’t all targeting HFT or other financial services.

> targeting HFT or other financial services

And the hyperscalers.

The Googles of this world only really build their own data centres in the US and maybe, if you're lucky, one or two other places elsewhere.

But generally, everywhere else in the world, and absolutely certainly in London, they're in the same buildings as everyone else ... except they rent entire floors and not just a few racks of course !

> Optimally you'd want a distributed CDN, but that doesn't really sound feasible without the cloud

CDN is a good choice when you ether need to sever small volume (i. e. not 500Tb) with low latency, e. g. to reduce web page load time by hosting static assets (js/css/img) close to the customer or when you have really huge traffic (e. g. Netflix, but they decided public cloud is not cost effective for CDN and built their own).

For NixOS mirrors latency is not important, traffic is not huge (withing capacity of a single commodity server) and just two mirrors - in US and EU would be close enough to majority of users.

I would split it in pieces and create a DIY CDN (like lots of Linux distros have). From the numbers I've seen in the article, the big problem is the storage, not the transfer, although they say that the space can be reduce by garbage collecting the less used caches (although not ideal).

Note that they are using Fastly as CDN in the front, but that's not important for them (monetary wise at least). The big part of the bill is just the storage part.

With little detail about their data I would chose one of these architectures...

  Debian (or other distro) mirror style:
  - Split the cache storage in pieces (by architecture, release or other categories)...
  - Deploy a bunch (10-30) of cheap servers in affordable hosting around the world. The most popular data is distributed in more servers (for example, the x86_64 build caches for the latest version of the packages)
  - Get some people to volunteer with small mirrors

  CDN style:
  - Get a couple or three medium/big sized (in disk, not CPU) servers for storing all the cache, for example, with Minio, so they don't even have to stop using S3 storage libraries.
  And then either:
  - Keep using Fastly
  or
  - Distribute tiny servers with Varnish or the like
  - Get some people to volunteer tiny edge cache servers

Some weeks ago this was on the front page:

- Serving 90TB/Day of Linux Updates from Thin Clients: https://news.ycombinator.com/item?id=35883053

- The scripts they use: https://github.com/PhirePhly/micromirrors

Surely you know what colocation is, don't you? Suggesting someone has to build a datacenter for it is like saying that programming is time consuming because people have to write their own compilers.

If someone asked me to do this, I'd colocate in two locations (likely Europe and west coast USA) two full copies of everything in 1/4 rack. After physically building, testing, installing, then testing some more, which likely would take a month, it'd be a part time job to monitor and babysit for the first few months of use, then it'd be an hour a month maintenance job.

CDNs cost money and come with tons of problems. For non-interactive static files, they're not necessary.

If it requires constant attention, you're doing it wrong :)