News for hackers. Stuff that matters.
Re:Secrets of PostgreSQL Performance (DjangoCon) by bmurphy on Wednesday September 21, 2011 @10:44PM (#3024638)
Very nice, thank you. Mostly all of this is good advice and correlates nicely with my experience.

One thing though:

"EBS volumes and Software RAID is best but scary on AWS"

I've managed an EBS RAID10 database for a few years now. I wouldn't touch this with a 10 foot pole.

Do yourself a favor, set up an m1.xlarge (or bigger) instance, put the ephemeral drives in a RAID0 and mirror across multiple machines using hot-standby, slony, londiste, or some other tool. You'll be much happier, your system will perform much better, and you'll have a failover strategy in place.

Re:Secrets of PostgreSQL Performance (DjangoCon) by sandGorgon on Thursday September 22, 2011 @01:34AM (#3024990)
I dont understand this - is there a difference between an explicitly allocated EBS volume and the ephemeral volumes of an EBS-backed instance ?

Or are you focusing on the RAID10 part - but then everywhere RAID10 is touted to be the best RAID solution (right balance between performance and safety)

Re:Secrets of PostgreSQL Performance (DjangoCon) by bmurphy on Thursday September 22, 2011 @07:34AM (#3025667)
EBS is a shared networked SAN. The performance characteristics of it are not that great and even worse, highly variable. The last thing you want to be running your database on is a system where the performance characteristics vary greatly throughout the day and you have no control over it.

The ephemeral drives are drives directly attached to the server and to the best of my knowledge are not a shared resource. Their performance characteristics are highly consistent, but if your server goes down all data on those drivers are lost.

EBS sounds nice in theory, but by going to EBS RAID you throw away most of its benefits (such as snapshotting) and take on it's worst aspects.

Re:Secrets of PostgreSQL Performance (DjangoCon) by alnayyir on Thursday September 22, 2011 @12:58PM (#3026735)
Ephemeral volumes are shared, just not nearly as variable.
Re:Secrets of PostgreSQL Performance (DjangoCon) by joevandyk on Thursday September 22, 2011 @03:33AM (#3025225)
Yes, there is a difference between EBS volumes and ephemeral volumes of an EBS-backed instance.
Re:Secrets of PostgreSQL Performance (DjangoCon) by jdunck on Thursday September 22, 2011 @12:58AM (#3024914)
And what happens when the AZ goes down? Don't you lose data then? Or do you mean multiple machines across AZ/region?
Re:Secrets of PostgreSQL Performance (DjangoCon) by joevandyk on Thursday September 22, 2011 @03:31AM (#3025216)
You'll want the mirror in a different AZ. Or you could also put it in a different region.

Use Wal-E to push wal logs to s3.

Re:Secrets of PostgreSQL Performance (DjangoCon) by bmurphy on Thursday September 22, 2011 @07:35AM (#3025671)
Absolutely. You should always be spreading your data across multiple availability zones and where feasible across multiple data centers and S3 is a great place to store your wal logs. We do the same thing.