News for hackers. Stuff that matters.
Re:Redis as the primary data store by roskilli on Monday April 08, 2013 @10:33AM (#5511894)
I'd be really interested in seeing if you benchmarked fsync at every query vs your fsync every second policy.

My main beef with fsync every second is just that you will never, ever get a "this is what the server looked like when it went down". If the fsync at every query was only worse by a relatively small factor, and if you used write transactions for the majority of your writes (meaning fewer times needed to fsync for every query) which I'm guessing you are to protect integrity on writes, I don't see why this wouldn't be more appealing than fsync at every second?

Re:Redis as the primary data store by anveo on Monday April 08, 2013 @11:58AM (#5512490)
Both of those settings might be somewhat problematic with stock EBS volumes. We run everysecond and very frequently see "Asynchronous AOF fsync is taking too long" because EBS can't keep up. The problem is when that happens Redis is blocking connections and exceptions pop up from the clients.

A work around so far is to sync every 60 seconds on the master, and more frequently on the slaves. Another option might be to bump up the IOPS on the volume, but I believe that still isn't available on medium instances (which we are using as well).

Re:Redis as the primary data store by devd on Monday April 08, 2013 @05:37PM (#5514805)
Just use the instance storage instead of EBS. Then write a script to move the AOF file from instance storage to EBS/S3
Re:Redis as the primary data store by courtneycouch0 on Monday April 08, 2013 @10:44AM (#5511950)
To be honest, we have not played with fsync every second. Redundant persistence servers across availability zones gives me enough comfort that my worry about losing that 60 seconds of data is rather low.

I suspect we will play with this over time and honestly you have me curious now too what the throughput differential would be for individual persistence servers.