Yup, but cost is relative to the problem you're trying to solve.

When your app needs horizontal scaling DB with multiple regions, and you don't have to come up with your own sharding solution, suddenly $8000 a year sounds like a great price.

General computer science-y kind of software philosophy question:

Should we really be thinking of sharding as something you can outsource to the infrastructure layer?

It just seems like having a stance about how the data in your specific domain naturally shards is probably going to pay off not even the long run, but immediately in terms of sanity checking your information architecture.

And I wonder if in 2017 we haven't gotten to the point where a cloud application should just shard, because we're trying to think about software as something that runs on a transient instance with access to a subset of data, appearing and reappearing, not a giant box with everything on it.

I get that Google engineers are darn close to abstracting away that "giant box with everything on it" behind an API, but I guess I'm asking if that's really how we should be thinking about our code.

As with basically all software design questions, the answer is "it depends".

Writing a LOB app with 4 expected users all located in the same location, and expected storage requirements of a gigabyte a year? Go with a traditional DB, add replication if you need high availability, make and test your backups, done.

Writing the new Facebook weknoweverything app that captures smartphone audio and video continuously for all users at all times, automatically transcribes all conversations and produces AI-driven summaries of all video, all of which are saved forever and highly searchable? You're going to need a highly customized data storage architecture to have a prayer of keeping up with that data.

Somewhere in between? Some of those are good candidates for Spanner. Some aren't.

I agree with your line of thinking. But at really big companies with hundreds of small teams, all working on separate projects, when the boss says to hook all that stuff together I'm sure a solution like Spanner looks really delicious and cost-effective.

For you and me working with data we already understand and added to our project piece by piece, we can safely shard and compartmentalize.

They're effectively competing with the likes of NonStop doing linear scaling on fault-tolerant systems but geographically distributed. The mainframe and NonStop solutions start at $1+ million if volume is significant. The VMS clusters can similarly cost 6 digits. Spanner might be cheaper than them if you need availability, consistency, and speed more than space.

I guess I was looking at it from the wrong perspective then. It is very expensive for me then :)

I guess it is not even meant for me (a single developer with a small project) then!

I hear you on that. It's why I'm holding high hopes for CochroachDB. Otherwise, ain't no chance I can have this kind of DB. However, there is opportunity for strong consistency in a cluster in small area just distant enough to reduce cascading failures. A project might build a VMS-style, clustering system for that integrated with FOSS database. Could exist but DB subfield is too flooded for me to track.

Also take a look at https://www.pingcap.com/ for sql scale-out and http://www.scylladb.com/ for cassandra-but-fast.

Thanks for reminding me about TiDB. Turns out I was on the first thread here about it. The next had some interesting comments:

https://news.ycombinator.com/item?id=13298664

Particularly, the comments about them mixing Go and Rust wondering if they should move to gRPC. The integration complexity concerns me a bit but at least they're smart about what pieces they use. There was also a lot of detail in the architecture although consistency and availability vs Spanner was not clear in my links. I'll have to look into it more. Thanks.