News for hackers. Stuff that matters.

A comment on Etsy is under pressure to start acting more like a conventional company. Parent comment

Re:Etsy is under pressure to start acting more like a conven... by nasalgoat on Thursday May 18, 2017 @08:16PM (#14372118)
We have billions of rows of data and very complicated CTEs doing joins on a dozen tables matching based on GEOS radius data, and it's the primary query for our application. We're running PostgreSQL 9.5 and need to have all the data in memory for the fastest results.

Part of the problem is that the network disk that EC2 provides as EBS is 100x slower than local disk, so keeping all the indexes and data in memory is the only way to replicate physical hardware performance.

If anyone knows of a better EC2 setup for PostgreSQL, I'm all ears.

[ Parent | on HN ]
Re:Etsy is under pressure to start acting more like a conven... by anarazel on Thursday May 18, 2017 @09:08PM (#14372341)
> Part of the problem is that the network disk that EC2 provides as EBS is 100x slower than local disk, so keeping all the indexes and data in memory is the only way to replicate physical hardware performance.

> If anyone knows of a better EC2 setup for PostgreSQL, I'm all ears.

Have a proper replication & archiving setup, and use instance storage. If you have configured streaming replication to 1-2 other servers, and archive your WAL to s3 (using wal-e or such), you're already above EBS's guarantees (99.9% durability IIRC?).

[ Parent | on HN ]
Re:Etsy is under pressure to start acting more like a conven... by nasalgoat on Friday May 19, 2017 @10:27AM (#14375937)
Instance storage is much more limited and is heavily tied to the specific instance types you pick.
[ Parent | on HN ]
Re:Etsy is under pressure to start acting more like a conven... by anarazel on Friday May 19, 2017 @01:18PM (#14377657)
Indeed. I didn't want to say it's a panacea, just that you can sometimes get a lot better performance for your money. E.g. larger i3's both have decent IO performance and sizes.

Still doesn't even remotely compete with what you can get with "normal" hardware.

[ Parent | on HN ]