News for hackers. Stuff that matters.
Re:Shopify Ruby on Rails distributed monolith runs 19M queri... by fardinahsan on Wednesday November 29, 2023 @02:17AM (#38456512)
Can you go in some high level details as to why it was slow and what you did to make it fast. That's always the most interesting part of a post like this.
Re:Shopify Ruby on Rails distributed monolith runs 19M queri... by kawsper on Wednesday November 29, 2023 @04:21AM (#38457269)
I've done something similar, you start out with something naive and simple like mysqldump that takes an age, and move on to more specialised tools like Percona XtraBackup that allows for incremental backups.
Re:Shopify Ruby on Rails distributed monolith runs 19M queri... by forgingahead on Wednesday November 29, 2023 @05:57AM (#38457898)
Any tips on similar tools for Postgres?
Re:Shopify Ruby on Rails distributed monolith runs 19M queri... by __s on Wednesday November 29, 2023 @08:34AM (#38459129)
Depends on environment. If you can do disk snapshots that's the way to go (can be hard with disk striping). wal-g works for storing both base backups & wal to various storages in parallel & can be throttled with env variables

Source: worked on Azure's managed Citus pg

Re:Shopify Ruby on Rails distributed monolith runs 19M queri... by firefoxd on Wednesday November 29, 2023 @11:34AM (#38461572)
The db connections were poorly managed. Each query started by starting a new connection. It then checked if it failed, slept, retried, then ran. There were several queries called in loops, so the connection pool was always dry. The code spent most of it's time sleeping. Then the queries themselves were highly inefficient, bad joins, no indexes, etc. It was satisfying to fix the mess.
Re:Shopify Ruby on Rails distributed monolith runs 19M queri... by dunno7456 on Wednesday November 29, 2023 @06:02PM (#38466560)
I had two similar experiences in the past years: I added some database indices, the application become super fast, the rest of the team starts acting weird, I get shown the door.

At this point I'm pretty sure anything IT relate became a bullshit job.