Re:Wal-E: Continuous Archiving for Postgres
by sdrothrock on Sunday February 05, 2017 @07:37PM
(#13575647)
> The best part is, you'd do a COMMIT and it would essentially block until the data was in HDFS.
Do you have any problems with freezes or timeouts during high loads?
Re:Wal-E: Continuous Archiving for Postgres
by ninkendo on Sunday February 05, 2017 @08:05PM
(#13575773)
We had freezes/timeouts only when the disk filled up, other than that it was pretty good. Simply writing the data to HDFS is a lot cheaper operation than committing the data to a local database, so in practice it was a lower latency than having a "real" standby server that committed logs to disk. (We saw about a 3x TPS gain over a dedicated standby in synchronous mode.)
Also the way I see it, turning on 'synchronous_standby_names' was a nice added safety guarantee, in practice leaving the hdfs receiver asynchronous would be a reasonable alternative if you're confident things will work as you expect.
Where I was, we investigated wal-e and determined it doesn't do anything that the raw curl commands wouldn't be able to do for us.
What we decided to do was to utilize HDFS for continuous archival, and we patched a version of `pg_receivexlog` to actually stream the WAL files out to HDFS, with flushing support that acknowledged transactions over the wire when the flushes were complete.
With this, you could treat this patched pg_receivexlog_hdfs command as a standby postgres database, and even add it to `synchronous_standby_names`, and postgres would effectively have its transaction logs synchronously written out to durable storage. We combined it with a lot of plumbing and we were able to get postgres running in mesos with docker without any actual persistent storage volumes (just used local disk.)
The best part is, you'd do a COMMIT and it would essentially block until the data was in HDFS. No periodic snapshots where you'd lose transactions that happened seconds after the snapshot... if the client sees a transaction as committed, it's on durable storage.
Worked pretty well, I'm surprised WAL-E doesn't support something similar (it only checkpoints at predefined intervals, not on transaction commit time.)