For over a hundred nodes? A lot, but these aren't little VMs with 4G of RAM and two vCPUs we are talking about there. Almost nobody has databases that big.

For a more realistic deployment in the cloud with 2TB of data, 4TB of backup storage and peak of 64 cores with elastic scaling it's about 6k/month. So the moment you're spending more than a third of a decently skilled SWE on working around database limitations it's worth it. That's for a fully managed HA cluster with 99.95% uptime, rolling upgrades etc.

> 2TB of data, 4TB of backup storage and peak of 64 cores with elastic scaling it's about 6k/month

right, but foundationdb and similar is for cases where you actually need throughput of hundreds servers, for what you described OSS Postgres will work well.

It's confusing because of terminology differences. What FoundationDB calls a "server" is a single thread, often running on a commodity cloud VM. What Postgres calls a server is a multi-process instance on a single machine, often a commodity cloud VM. What ExaData calls a server is a dedicated bare metal machine with at least half a terabyte of RAM and a couple hundred CPU cores, with a gazillion processes running on it, linked to other cluster nodes by 100Gb networks. Often the machines are bigger.

Obviously then a workload that might require dozens or hundreds of "servers" in FoundationDB can fit on just one in an Oracle database. Hundreds of nodes in an ExaData cluster would mean tens of thousands of cores and hundreds of terabytes of RAM, not to even get into the disk sizes. The nodes in turn are linked by a dedicated high-bandwidth network, with user traffic being routed over the regular network.

As far as I know, nobody has ever scaled a FoundationDB cluster to tens of thousands of servers. So you can argue then that ExaData scales far better than FoundationDB does.

This makes sense for the design because in an RDBMS cluster the inputs and outputs are small. A little bit of SQL goes in, a small result set comes out (usually). But the data transfers and work required to compute that small result might be very large - in extremis, it may require reading the entire database from disk. Even with the best networks bandwidth inside a computer is much higher than bandwidth between computers, so you want the biggest machines possible before spilling horizontally. You just get much better bang-for-buck that way.

In FoundationDB on the other hand nodes barely communicate at all, because it's up to the client to do most of the work. If you write a workload that requires reading the entire database it will just fail immediately because you'll hit the five second timeout. If you do it without caring about transactional consistency (a huge sacrifice), you'll probably end up reading the entire database over a regular commodity network before being able to process it at all - much slower.

> Postgres calls a server is a multi-process instance on a single machine, often a commodity cloud VM. What ExaData calls a server is a dedicated bare metal machine with at least half a terabyte of RAM and a couple hundred CPU cores, with a gazillion processes running on it, linked to other cluster nodes by 100Gb networks. Often the machines are bigger.

You can also have PG cluster of beefy machines linked by 100Gb network, I didn't get what's the difference in this case.

This is again an issue of terminology. A PG cluster is at best one write master and then some full read replicas. You don't necessarily need a fast interconnect for that, but your write traffic can never exceed the power of the one master, and storage isn't sharded but replicated (unless you host all the PG files on a SAN which would be slow).

A RAC cluster is multi-write master, and each node stores a subset of the data. You can then add as many read caches (not replicas) as you need, and they'll cache only the hot data blocks.

So they're not quite the same thing in that sense.

PG ecosystem actually has multiple shared nothing cluster implementations, example is www.citusdata.com, unlike RAC, where masters need to sync to accept writes, so technically write load is not distributted across servers.

Citus is great but it's just sharding. Only some very specific use cases fit within those limits, and there are cost issues too (being ha requires replicas of each node etc). RAC masters don't need to sync writes to each other, that's not how it works. Every node can be writing independently, they only communicate when one node has exclusive ownership over a block another node needs to write to, and the transfer then occurs peer to peer over rdma. But if writes are scattered they work independently.

> they only communicate when one node has exclusive ownership over a block another node needs to write to

and it needs to communicate that he has exclusive ownership, and also after each writes you need to invalidate cached data on other nodes.

> fully managed HA cluster

This is where Suspension of Disbelief stops for me — I've been taught by years of Jepsen analyses that "HA" really doesn't exist, especially in the SQL database world, and especially if there is a big company behind with lots of big complicated buzzwords.

What you usually get is single master replication, and you get to pick up the pieces if the master dies.

Jepsen doesn't work through databases in any kind of order as far as I can tell, and they haven't done an analysis of Oracle or many other popular databases. So I wouldn't take that as a representative sample.

RAC clusters are multi-write-master and HA. The drivers know how to fail over between nodes, can continue transactions across a failover and this can work even across major version upgrades.

I wonder if the clause in the Oracle license that prohibits publishing benchmarks and other studies could be to blame for Jepsen having not investigated..?