Now, one approach is to just dismiss this use-case by pointing at DynamoDB and similar offerings. But if for some reason you can't use these hosted platforms, what do you use instead?
For search, ElasticSearch fortunately fits the bill, the "just keep adding boxes" concept works flawlessly, operating it is a breeze. But you probably don't want to use ElasticSearch as your primary datastore, so what do you use there? I had terrible experiences operating a sharded MongoDB cluster and my next attempt will be using something like ScyllaDB/Cassandra instead since operations seem to require much less work and planning. What other databases would offer that no-advance-planning scaling capability?
Somewhat unrelated, by I often wonder what one were to use for a sharded/distributed blob store that offers basic operations like "grep across all blobs" with different query-performance than a real-time search index like ElasticSearch. Would one have to use Hadoop or are there any alternatives which require little operational effort?
Not even transparent pricing?
It’s admittedly not at all obvious from the repo name, but the following is the top level repo for Couchbase Server (which uses Google’s `repo` tool): https://github.com/couchbase/manifest
Geez, with all these limitations you may as well use another, proper open source database.
What kind of data and amounts do you frequently encounter that is a good fit for relational storage but difficult to fit in a single server? Honestly curious.
And you say update... Across keys, or some keys depend on others? Otherwise it sounds like a perfect fit for sharding with reduced constraints/requirements.
You can even use the Foundation Document Layer which is API compatible with Mongo.
It definitely takes some getting used to, but I think it's pretty fucking great, once you do.
CouchDB is re-architecting onto it, https://youtu.be/SjXyVZZFkBg
I think the biggest reason for a lack of noise about it, is the overhead of learning it is pretty high. You're not going to find folks writing their first "Nodejs + Express" applications using it. Additionally, you really have to know why most distributed databases suck ass to know why FoundationDB is so good.
Example: I have a cluster of three VMs running FDB on my home server, and over the past week I've accidentally hit the power switch three or four times. At no point did I have data loss in the cluster, the cluster "immediately" comes back up, and is ready to go. Adding machines to the cluster is unbelievably easy, especially if you have ever even tried grokking how to horizontally scale PSQL.
I'm close to releasing an Elixir-based Entity Layer which is pretty uhhh, ~~shitty~~ lacking, at this point feature wise, but it does make storing structured data a bit easier. I'm hoping it'll be more useful for helping people learn how to use FoundationDB than something folks are putting into production (though I'm dog-fooding it).
Yes, but it's also a wonderful way to corrupt a DB as soon as anything goes wrong in your storage fabric system.
And this type of corruption is generally not the recoverable one.
Why is this more likely to corrupt a DB than having it on local disks, when something goes wrong?
Local filesystems (ext4, xfs) have been designed to run over reliable internal bus and over spinning disk. They are (almost) able to stand most of the outage happening in this context ( powercut, corrupted block, missing flush ).
Put them over a non-reliable "normal" network, where you can get "savage" cable unplugged, faulty controller, paquet loss, out of order delivery, buggy middle box and you explode the number of scenario that can go wrong and will go wrong.
Block level I/O virtualization is amazingly useful, but (in my mind) it should be used with care...
I already heard of case in prod where the Master DB, the Slave DB and backup finished all in the same virtualized block-level storage.... try to guess what happened next.
Plus, a single server with many TB is a big single-point-of-failure, and if you want to scale it (vertically) you still have to take it down.
But you can still make this work. In our case, we'd ended up going the first route, and we ended up adding AWS EBS as the filesystem block store for our Postgres database, which is easy to resize dynamically without incurring downtime or other issues.
The downside of a vertical scaling approach is, well, you don't get HA "for free". You have to manually configure followers and standby nodes for the sole machine. You have to worry about your own replication. If failover and management is abstracted out for you, as in RDS, then it's easy to live with - otherwise very painful.
tl;dr go with managed DBs if you just need a DB and don't particularly have to optimise query performance outside of DB config.
There was a really good post from Martin Fowler a while back that the popularity of "NoSQL" was really because it was "NoDBA" - app devs could sidestep the bottleneck of needing to get DBAs involved whenever you needed to persist an extra object field. While it's easy to abuse JSON storage in postgres, for things that are really just "opaque objects", vs. relational properties, appropriate use of JSON columns can save a ton of unnecessary overhead updating schemas.
I think Scidb, being a column-oriented dbms geared for multi-dimensional arrays (datacubes) is very interesting given current trends, and there are only a handful of similar dbs around, the other two that interest me are rasdaman and monetdb. I don't know if OmniScidb counts as a datacube db but it is also really interesting especially due to it's gpu and caching model.
As a sysadmin/ops type having to deal with monitoring timeseries db's are also something I like to keep an eye on. It used to be mostly rrdtool in this space, but now I am comparing prometheus and influxdb. Like OmniSci, another case of something that's not quite in the same db model but might even better a better solution for the space (metrics) is Apache Druid. (elasticsearch being another than can be massaged to fit as a quasi tsdb as well) I think there is some room to unify the monitoring/metrics and log storage arena into one space (usually they are separated which adds admin overhead) and right now I really like druid as a potential for this.
Another interesting application of timeseriesdb's that I have been keeping an eye on is in the quant/algo trading area. Most people have been using kdb+ there but many are looking for replacements and there are some really good conversations to be found about the kind of limitations they are hitting.
I'm just a sysadmin who likes to keep up with whats going on, and my db knowledge is limited, but I do have a process for narrowing my focus to the dbs I reference. It must be open source, bonus points to gpl or apache licenses. The language it is written in is important but not a deal breaker (very tired of so many java based dbs). I don't like it when they are tacked on top of an "older" tech (such as kairos on cassandra, timeseriesdb on top of postgres, opentdsb on top of hbase, kudu on hadoop, etc). Being either filesystem aware or agnostic can be nice features (playing well with ceph, lustre, etc) Not saying this is the sort of selection criteria others should use just giving some info on mine.
A few more interesting mentions: clickhouse, gnocchi, marketstore, Atlas (Netflix), opentick (on top of foundationdb).
Performance is good but comes at a massive usability cost. kdb+ brag about very succinct commands and even error messages but the small gain in efficiency from one less character in the error message is at the detriment of the user trying to work with it.
QuestDB (http://questdb.io) is an alternative to kdb as time-series data store (disclosure, i am one of the authors). We left trading to build it and show you can get the same speed without sacrificing usability. Users get better performance than kdb+, but can use SQL, ACID transactions. It is already in production with a few exchanges and HF, and it is open-source!
Hopefully, kdb users looking for a replacement find questdb helpful for their projects :-)
RethinkDB, CockroachDB and FoundationDB are worth keeping an eye on.
Yeah, MySQL is ahead of the curve there...
Still, I can only be carefully optimistic.
However, I do think the key innovations are in building control planes around existing relational and NoSQL databses for scaling/sharding them across a set of resources to minimize cost while meeting performance and availability constraints.
In what way was it interesting? It was a document DBMS that supported MVCC.
They really shine for read heavy workflows that can tolerate a stale read every once in a while. If on top of that you have reasonable shard-ability you get near infinite scalability.
While that might cover a large portion of the database usage landscape, I'd hesitate to call it most. There's a reason OLTP was coined as an acronym--it's a pattern that comes up a fair bit.
Netezza, Informix, KDB and others will all well outperform open source dbs.
All open source dbs are absolute trash for time series an other olap type queries.
edit: after looking it up again, looks like that is still the case, and you have have to be fairly limited with cumulative aggregates if you want the keep performance. maybe someday, but as of now, still not very good.
It does make me wonder what will be the next big leap in DB technology. Most of the NoSQL or distributed DB implementations have a bunch of limitations which make them impractical (or not worth the trade offs) for most applications, IMO. Distributed DBs are great until things go wrong, and then you have a nightmare on your hands. It's a lot easier to optimize simple relational DBs with caching layers, and adding read replicas scales quite effectively too.
The only somewhat recent new DB that comes to mind which had a really interesting model was RethinkDB, although it suffered from a variety of issues, including scale problems.
Anyway, these days I stick with Postgres for 99% of things, and mix in Redis where needed for key/value stuff.