News for hackers. Stuff that matters.
Re:We scaled the GitHub API with a sharded, replicated rate ... by gigatexal on Thursday April 08, 2021 @11:51AM (#26740277)
We had a saying at my old job: if something’s broken it’s never Redis. Redis is such a tank in my experience. We set it up. Secured it. And then forgot about it.
Re:We scaled the GitHub API with a sharded, replicated rate ... by junon on Thursday April 08, 2021 @11:58AM (#26740365)
Absolutely. Of course I plan for what happens if Redis were to fail, but I don't remember a single time it ever did. It really, truly is a tank.

Antirez, if you're reading this (I know you'll eventually find your way here), thanks for making my job a little easier over the years <3

Re:We scaled the GitHub API with a sharded, replicated rate ... by rshaw1 on Thursday April 08, 2021 @12:53PM (#26741059)
I thought the same up until yesterday!

Yesterday the replica randomly disconnected from the master and could no longer reconnect, the resyncs were failing because a replication buffer on the master was being hit (https://redislabs.com/blog/top-redis-headaches-for-devops-re...). Once that buffer was increased the replica was able to sync the snapshot from the master but for some reason was taking a very long time to load the snapshot, during this time the sentinels thought the replica was healthy again and started allowing the application to read from them, of course Redis responded with an error "dataset loading". We were running Redis 6.0.8 and upgrading the replica to 6.2.1 allowed the replica to sync and become healthy in seconds.

I'm still not sure why the sentinels thought the replica was healthy as issuing commands to it always returned an error.

Re:We scaled the GitHub API with a sharded, replicated rate ... by gigatexal on Thursday April 08, 2021 @12:55PM (#26741089)
Whoa! You should blog about this for posterity for the next hopeless soul who has to figure out why the thing that never dies died.
Re:We scaled the GitHub API with a sharded, replicated rate ... by lefrancaiz on Thursday April 08, 2021 @01:28PM (#26741471)
>The sharding project bought us some time regarding database capacity, but as we soon found out, there was a huge single point of failure in our infrastructure. All those shards were still using a single Redis. At one point, the outage of that Redis took down all of Shopify, causing a major disruption we later called “Redismageddon”. This taught us an important lesson to avoid any resources that are shared across all of Shopify.

It seems that it does happen sometimes, however.

https://shopify.engineering/e-commerce-at-scale-inside-shopi...

Re:We scaled the GitHub API with a sharded, replicated rate ... by spullara on Thursday April 08, 2021 @04:22PM (#26743299)
At Twitter we hit 15s memory allocation pause times due to fragmentation. We had to switch the memory allocator to fix it.
Re:We scaled the GitHub API with a sharded, replicated rate ... by hrpnk on Thursday April 08, 2021 @04:50PM (#26743605)
Which memory allocator did you end up using?
Re:We scaled the GitHub API with a sharded, replicated rate ... by spullara on Thursday April 08, 2021 @05:00PM (#26743699)
I think we settled on jemalloc but my memory may be failing me.
Re:We scaled the GitHub API with a sharded, replicated rate ... by lenn0x on Friday April 09, 2021 @10:23AM (#26750944)
You are correct that's what it was
Re:We scaled the GitHub API with a sharded, replicated rate ... by o-__-o on Thursday April 08, 2021 @10:28PM (#26746091)
I see what you did there