Out of curiosity, why not go bare metal in a managed colocation? Is that for the geographic spread? Or unpredictable load?

Every few months of this spend is like buying a server

Edit: back at my pc and checked, relevant bare metal is ~$500/m, amortized:

https://baremetalsavings.com/c/LtxKMNj

Edit 2: for 100tb..

Hiring someone who knows how to manage bare metal (with failover and stuff) may take time %)

You pay a datacenter to put it in a rack and add connect power and uplinks, then treat it like a big ec2 instance (minus the built-in firewall). Now you just need someone who knows how to secure an ec2 instance and run your preferred software there (with failover and stuff).

If you run a single-digit number of servers and replace them every 5 years you will probably never get a hardware failure. If you're unlucky and it still happens get someone to diagnose what's wrong, ship replacement parts to the data center and pay their tech to install them in your server.

Bare metal at scale is difficult. A small number of bare metal servers is easy. If your needs are average enough you can even just rent them so you don't have capital costs and aren't responsible for fixing hardware issues.

sounds like an opportunity for someone (you?) to offer an abstraction slightly above bare metal to do the stuff you said to do, charging higher than bare metal but lower than the other stuff. how much daylight is there between those prices?

I'm sure there are companies in this space providing private clouds on bare metal, I wonder how that would be to operate at scale though.

We run on our own stuff at our shop.

Some things that are hidden in the cloud providers cost are redundant networking, redundant internet connection, redundant disks.

Likely still cheaper than the cloud obviously but you will need to stomach down time for that stuff if something breaks.

Are you going to risk your entire business over "probably never get a hardware failure" that, if it hits, might result in days of downtime to resolve? I wouldn't.

Just pay 2x for the hardware and have a hot standby, 1990s-style. Practice switching between the boxes every month or so; should be imperceptible for the customers and a nearly non-event for the ops.

How many hours of labor does that take every month you failover? What about hot hard drive spares? Do you want networking redundancy? How about data backups? Second set of hot servers in another physical data center?

All of that costs money and time. You're probably better off using cloud hosting and focusing on your unique offering than having that expertise and coordination in house.

agreed, one month of 50 TiB is $12,800!

we're using Filestore out of convenience right now, but actively exploring alternatives.

This is compelling but it would be useful to compare upfront costs here. Investing $20,000+ in a server isn't feasible for many. I'd also be curious to know how much a failsafe (perhaps "heatable" cold storage, at least for the example) would cost.

Yes, you should be in our target market. I don't think that I can give a cost estimate without having a good sense of what percentage of your data you're actively using at any given time, but we should absolutely support the performance numbers that you're talking about. I'd love to chat more in detail, feel free to send me a note at hleath [at] regattastorage.com.

I'll send you a note!

Found this in the docs:

> By default, Regatta file systems can provide up to 10 Gbps of throughput and 10,000 IOPS across all connected clients.

Is that the lower bound? The 50 TiB filestore instance has 104 Gbps read through put (albeit at a relatively high price point).

That's just the limit that we apply to new file systems. We should be able to support your 104 Gbps of read throughput.