News for hackers. Stuff that matters.

A comment on Jepsen: Testing Partition Tolerance of PostgreSQL, Redis, MongoDB, Riak (2013). Parent comment

Re:Jepsen: Testing Partition Tolerance of PostgreSQL, Redis,... by falcolas on Friday May 02, 2014 @11:34AM (#7686302)
From the point of view of a "devops" developer - who has done hard time in QA and DBA as well - this requirement changes drastically when performance and scaling enter the equation. The scale slides quickly from "lose no data, ever" to "we can lose a few seconds of transactions if it speeds up the web page" to "we can lose a lot of data and still be OK, as long as we're still online".

Seconds, or even minutes worth of lost data do cost the company, but not nearly as much as poor performance. And unfortunately, developers tend to over-value data in the equation, leading to decisions which cause a company problems when it comes time to grow.

Ultimately, the best tools for ensuring business continuity (with few exceptions) is redundancy coupled with a set of proper backups.

[ Parent | on HN ]
Re:Jepsen: Testing Partition Tolerance of PostgreSQL, Redis,... by olavgg on Friday May 02, 2014 @11:50AM (#7686399)
Wouldn't message queues help you with this? They can be easily scaled horizontally to handle any write load and correctly insert data to the database safe over time.

You can also shard your database to distribute write load.

[ Parent | on HN ]
Re:Jepsen: Testing Partition Tolerance of PostgreSQL, Redis,... by falcolas on Friday May 02, 2014 @12:09PM (#7686498)
All of those absolutely help, at the cost of added complexity, additional points of failure, and hardware/VPS costs. And you still risk losing data, or at least data integrity.

Not to mention there are still hard limits on how quickly you can insert data into a database with 100% durability (which is, of course, impossible, but another topic entirely), and there are scales where even these mitigation tactics can't help you anymore (in particular, online casino games have this problem since they are persisting the state of multiple players very frequently).

[ Parent | on HN ]
Re:Jepsen: Testing Partition Tolerance of PostgreSQL, Redis,... by mrmondo on Friday May 02, 2014 @06:45PM (#7688620)
Out of interest, What hardware have you tried running your Postgres clusters on?
[ Parent | on HN ]