Instead of using a map (or COMPACT STORAGE), they should have defined their schema upfront (one of the limitations brought on by not-SQL). However if they didn't want to do that then COMPACT STORAGE is obviously a better solution than using a Map.
To answer your question about COMPACT STORAGE, standard CQL basically does the work for you (in terms of parsing the "xsv") but you have you define the schema upfront (as in, you have know all the map keys before hand). The reason they tell you not to use COMPACT STORAGE is for those cases where you don't need a dynamic schema, using COMPACT STORAGE doesn't really get you anything.
Lastly, IMO, I wouldn't touch CQL Collections or COMPACT STORAGE unless absolutely necessary. If you do need a dynamic schema I would rather encode the data as a msgpack or protobuf blob.
We tried. It was not only less disk efficient, but also slower for queries. Wasn't the result we expected, but alas.
However, I'm even more surprised at the "slower for queries" part. Maybe I'll do some tests with COMPACT STORAGE.
Would have gladly gone into more detail, but at 4,000+ words for the blog post already... :)
I stand by my point that the mistake was using CQL maps when you should have just used a defined schema.
My concern with Redis is reliability. Redis Cluster has problems with consistency [1], whereas unclustered Redis -- well, it syncs to disk every 10 seconds or so, but even then I'm concerned that the reliability of its on-disk structures haven't been as battle-tested as, say, PostgreSQL. Or has it?
Yeah, about that:
http://redis.io/topics/persistence
> Using AOF Redis is much more durable: you can have different fsync policies: no fsync at all, fsync every second, fsync at every query. With the default policy of fsync every second write performances are still great (fsync is performed using a background thread and the main thread will try hard to perform writes when no fsync is in progress.) but you can only lose one second worth of writes.
If you are happy with a Master/Slave setup and occasionally having to deal with data loss due to Master failures [e.g. losing a second or two of data that wasn't replicated], it works nicely.
Just realize I wouldn't use Redis as a long term persisting of data because of issues like this:
Redis is neet if your information fits in RAM, but that's not what Cassandra is for.
You can easily get commodity servers with 256GB of RAM per node x 10 shards. Cluster or not, as per your use case.
row key = key
column = field
I know Lua is "an option" in Redis but I avoid it like the plague because under production loads, Redis hashes perform and are easy to maintain w/o Lua.
I've run multi-gigabyte data sets in Cassandra without any config changes. And when I responsible for a 40 node Cassandra cluster we didn't do any tweaks other than a few JVM settings here and there.
Cassandra I have to admit though is very sensitive to how you model and store your database. The whole tombstone saga is never a fun one to go through.
How were you querying 40-nodes or were you offloading the querying to Lucene or something [e.g. like Parsely is]?
WOW. You've run several GB of data through it? Wow.
Oh my. I look forward to your next "X is not a panacea" article on MongoDB.
Turns out commercial db vendors lie. A lot. Never use a feature until it's been in a couple point releases (y in x.y.z versioning). Upgrade slowly.
Also, Cassandra - and MongoDB when used with clustering - requires writing your data to best fit your queries. This often leads to wildly different schemas than you'd create in a traditional relational database.
You should probably read Kyle Kingsbury's article on MongoDB to have some idea of what you're really getting into: https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-read...
It's also almost mandatory to read the internal design docs of cassandra even if you're not the admin but just working with it. And modelling data is a lost less trivial than it looks - and almost always not what you assume.
Anyway, this is the best talk I've seen on Cassandra data modeling even if you know Cassandra but not on an expert level. I made sure everyone on my team saw it at least once. https://www.youtube.com/watch?v=qphhxujn5Es