Honestly, just use Spanner. It's the perfect database*. If you need to trade freshness for savings, put a cache in front of it.

If I were building a new startup in 2023, I would need a mountain of evidence against using Spanner. It's ugly that it locks you into GCP but hey an iPhone locks you into Apple's ecosystem, that's just the price you pay to get good things.

* unless you need timeseries, columnar, FTS, geospatial, graph or something special like that

Spanner is ridiculously expensive. Though I get your point, it’s not like an iPhone. An iPhone is not actually that expensive compared to top end android phones.

Spanner is very expensive even compared to hosted cockroach and RDS (Aurora). That being said I’m sure if you're enterprise it's substantially cheaper.

Spanner generally is about twice the cost as cockroach which seems cheap but the gap in nominal cost only grows with usage.

The other issue is that since there’s no source to self host you also have to pay, unlike cockroach.

Generally with cockroach I’d expect one to self host all instances except a staging/pre-pod and prod. With Spanner all your instances necessarily will be hosted, which means $$.

That being all sad, Spanner is worth it if money isn’t an issue

The thing is, there is nothing else like Spanner, unless you want to trust a startup with your database, which opens up Cockroach, Yuga, and TiDB. Databases like RDS don't compare because they don't offer the same guarantees. Nothing even in AWS' and Azure's portfolios is like Spanner.

Back of napkin math I did a couple of months ago: Spanner costs about $1/month for 2 writes/second and 10 reads/second. The minimum is $88 which is 176wps and 880rps. If your startup gets more traffic than that, you have good problems. The simplified guarantee model (fully linearizable and serializable everything; automagic sharding, replication and failover) could save you costs elsewhere.

Low-value high-volume data like logs and search queries can be sent to a cheaper self-hosted timeseries/fts database.

You forgot storage costs and especially bandwidth costs and lockin costs and probably range-queries costs.

I've built my own little pile of evidence against Spanner using it for several years, after previously being on Postgres. It has good potential but isn't there yet. And if you're making a startup, you're probably a long way from the scale where sharded DBs become necessary.

I hate this tired argument. If I'm building a startup I'm building it with the intention to hit scale. If I can use cockroach or yugabyte and get 90% of the benefits of postgres without significant additional cost I'm going to use it. I've been at multiple startups that have had trouble scaling their database and the cost to switch database technologies when you hit scale is massive. And worst of all, the limitations of the technology always impact product development. I don't care what cool features postgres has if I can't do online schema changes.

So yeah, I'd rather spend a few more hours picking an actual scalable technology now than spend what could be 7 figures and massively impact product development moving to scalable technology..

Different teams have different abilities. You can't really make general rules around any of this. Should you use spanner as a default selection for your company's database needs? Maybe.

both good points, including the devs having different abilities and using a ready-made scalable solution. We're living in the world of having something working that you can test in a few days shrug

Some general rules make sense for common use cases. Like, I'll say that a relational DBMS is generally what you want rather than a graph one. For a single-node DB, I'll always use Postgres if I have the choice.

But sharding is too use-case specific to prescribe something broadly like Spanner.

> If I can use cockroach or yugabyte and get 90% of the benefits of postgres without significant additional cost I'm going to use it

That's a big "if." Idk about those two, but you sacrifice a lot using Spanner. And in the initial phases, you can't tell what your actual needs will be at scale, so you'll probably have to rework things anyway. Just using Spanner doesn't solve your scaling problems.

Spanner is great if you have unlimited money

> perfect

Doesn't spanner introduce new (very unlikely) failure modes that other databases are not impacted by? The reliance on an external consistency model feels to me like a complete outsourcing of liability that warrants thorough investigation.

Hypothetically, if GPS went down for a prolonged period and/or a bug was found in the TrueTime system, what would happen to the consistency model around Spanner?

I feel like some applications and customers would much rather wait for a synchronous acknowledgement from the actual, live system. An extra 150ms when you are confirming a 6-figure wire transfer could easily be framed as a good thing in most circles.

There is synchronous acknowledgement. Spanner uses 2PC + paxos groups.

If there was a bug in TrueTime, you would susceptible to phantoms, which isn’t great but almost certainly not catastrophic.

I believe that’s the only isolation issue you’d hit, since spanner locks the rows involved in a transaction.

Spanner offers both linearizability and serializability. They use the non-standard term "external consistency" to describe the union of those two properties.

>Doesn't spanner introduce new (very unlikely) failure modes that other databases are not impacted by?

Curious if you could go into that.

Citus is not close to Spanner. No global secondary indexes, no cross-shard ACID (the commit status is eventually consistent). No auto-resharding. Yugabyte and Cockroach have a spanner-like architecture, but different open-source model and postgres-compatibility. What are those annoying issues?

This is true, but the article is talking about HA and Citus is HA, no?

by itself, Citus is just sharding to multiple databases. Then each of this database can be protected like any database with a standby

There are some fundamentally really hard problems with multi-node DBs, and the usable solutions depend a lot on your use case, so I'm not surprised at the current state.

Doesn't Spanner depend on access to globally-synchronized custom atomic clock hardware? i.e. even if it were open source, it wouldn't really be valuable unless you're a data center operator.

As I understand it, it's not a solved problem because there is no silver bullet, but rather trade-offs in every direction, and which solution works for you (including Spanner) is heavily dependent on your use case.

the clocks help but are not strictly required. FoundationDB also achieves external consistency without the clocks, but has limitations, such as a 5 second window for a transaction, due to optimistic concurrency.

the clocks remove such limitations.

From my understanding, the only advantage of atomic clocks is that you can horizontally scale the sequence number generator because all these atomic clocks can operate independently and yet produce the same numbers at the same time (basically by tapping into the fabric of the universe).

Out of curiosity, what's wrong with cockroach?