> And this most of the times does not happen too.

This rubs me very very wrong. There is no maybe in data integrity. Redis wasn’t designed to be ACID and shouldn’t be treated as such just because “usually” it doesn’t lose data “under the right circumstances.”

If a sysadmin uses an RDBMS with relaxed fsync, they know what they are getting (and durable writes is not one of them, no usually about it). Same for Redis.

Removing fsync is generally referred to as "running with scissors" mode, although postgres now supports it better with unlogged tables.

I came up with a "mental framework" that allowed me to get beyond these details early when I started architecting systems for companies.

1) Identify what are the possible malfunctions

2) What mechanisms will I use to detect these malfunctions as early as possible

3) Do I have a clear and concise recovery path for said malfunctions.

This frame of thought has taken me far (in terms of being able to accomplish a lot in my career so far) and allowed me to move forward without (a) extensive, time-consuming, research into the specific tools I'm using into the specific situations which may never happen, (b) extensive costs hiring experts in these niche areas.

Sometimes you don't need ACID guarantees, which is part of the reason databases like Redis exist, and is why the compromises you're describing are important. But in this context it sounds like the framework you're describing would lead to rolling your own ACID for Redis instead of just using an ACID database to begin with.

Redis was never designed for strong Synchronous replication of transaction or Strong leader election.

So each time a master crash you have a high chance of some recent acknowledged write being lost.

we live in the age of devOps. there's no such thing as sysadmin, or god forbid, DBAs.

/sarcasm

Favorite datastore to date. The only thing I wish for frequently is native global change-data-capture without doing something manually with streams (I've thought about writing a replication protocol consumer, but the way resync's are handled is a little problematic)

Worker is a DIY stream consumer implementation. The stream -> lambda integration works, but its super limited (and ~300ms slower to get new events on average)

Via the lambda integration you should probably expect ~1000ms p99 latency intra-region, and ~2100ms p99 from, e.g, east-1 to west-2. Shave maybe 200-300ms off of those numbers if you do a DIY stream consumer without the aws provided lambda connector.

> "Remember redis (in AWS) holds ephemeral data in memory"

(because EC2 machines don't persist their disk data between start/stops and a lot of people don't realize that until it is too late - hey remember that Bitcoin exchange that got bitten by this?)

EBS is now the default when creating an instance, and instance types of newer generations generally don't have instance store at all (except disk optimized instance types, and those with "d" in the name.)

Yes if you do an ordinary OS reboot the disk remains.

Bad idea. Bad tool for the job.

He sort of implies here, not for you, but it might create that impression on others, was clarifying for those.

I understand your defensiveness, but please understand that because of your massive experience with Redis, either you have never lost data with Redis, which means that nobody besides you understands how to run your software in production, or you have lost data with Redis, which means that it's somewhat hypocritical of you to insist that Redis is durable.

Hogwash and there are plenty of examples out there that prove otherwise.

> somewhat hypocritical of you to insist that Redis is durable

If Jepsen has demonstrated anything, it's that no database is as durable as it claims to be.

I can't think of a single database that's solved the fsync/O_DIRECT issue on Linux completely. Postgres had to patch it _again_ last year.

They always have bugs somewhere, but there are huge differences between bugs that show up for very specific, niche cases, and normal "I wrote to the db and it dropped it".

"We found five safety issues in version 1.1.1—some known to Dgraph already—including reads observing transient null values, logical state corruption, and the loss of large windows of acknowledged inserts."

Loss of large windows of acknowledged inserts. Durability is hard.

As staticassertion is mentioning, some of the violations that were found were only around tablet moves, which happen only in certain cluster sizes and quite infrequently. Of course, Jepsen triggers those moves left-right-and-center to evoke some of those failure conditions; but that's not how tablet moves are supposed to work in real world conditions. This is different from other edge cases like process crashes, or machine failures, network partitions, clock skews, etc., which can and do happen. In those cases, Jepsen didn't find any violations.

We were planning to look into those tablet move issues and get them fixed up (shouldn't be that hard), but honestly, the chances of our users encountering them is so low that we de-prioritized that work over some of the other launches that we are doing.

But, we'll fix those up in the next few months, once we have more bandwidth.

"All of the issues we found had to do with tablet migrations"

"ndeed, the work Dgraph has undertaken in the last 18 months has dramatically improved safety. In 1.0.2, Jepsen tests routinely observed safety issues even in healthy clusters. In 1.1.1, tests with healthy clusters, clock skew, process kills, and network partitions all passed. Only tablet moves appeared susceptible to safety problems."

No one is here to claim that anyone is getting through any kind of rigorous testing without bugs found. But there is a huge difference between "My extremely common write path + a partition = dropped transactional writes" and "Under very specific circumstances, with worst case testing, multiple partitions, and the db in a specific state, we drop writes".

There is an ocean between, say, mongodb's test results, and Dgraph's.

Read Redis's evaluation, for example: https://aphyr.com/posts/283-call-me-maybe-redis

"If you use Redis as a queue, it can drop enqueued items. However, it can also re-enqueue items which were removed. "

"f you use Redis as a database, be prepared for clients to disagree about the state of the system. Batch operations will still be atomic (I think), but you’ll have no inter-write linearizability, which almost all applications implicitly rely on."

"Because Redis does not have a consensus protocol for writes, it can’t be CP. Because it relies on quorums to promote secondaries, it can’t be AP. What it can be is fast, and that’s an excellent property for a weakly consistent best-effort service, like a cache."

Again, Redis is a very different type of database, so expectations should be aligned. Further, this test is quite old.

But that's a huge difference from DGraph's results.

Basically, saying "Well no one does well on Jepsen" isn't really true. Lots of databases do well, but you have to adjust your definition of "do well".

Redis is an extremely reliable service. I've never "lost" data with Redis.