Fly.io Postgres cluster down for 3 days, no word from them about it

Re:Fly.io Postgres cluster down for 3 days, no word from the... by kunley on Friday July 21, 2023 @11:44AM (#36815167)

> somehow a containerd boltdb on that host got corrupted, so when the machine bounced, containerd refused to come back online. We use containerd as a cache

Hey, even if I can feel sympathetic for the course of unfortunate events, it's hard to not to comment:

if you're using a cache, you should invalidate it on failure!

[ Parent | on HN ]

Re:Fly.io Postgres cluster down for 3 days, no word from the... by tptacek on Friday July 21, 2023 @11:55AM (#36815351)

It's a read-through cache. This wasn't a cache invalidation issue. It's a systems-level state corruption problem that just happened to break a system used primarily as a cache.

[ Parent | on HN ]

Re:Fly.io Postgres cluster down for 3 days, no word from the... by kunley on Friday July 21, 2023 @07:13PM (#36821279)

What I meant is that if the compromised host was unable to use broken boltdb cache, the cache should be zeroed and repopulated. Was it really hours of such cache rebuild vs hours of trying to fix the boltdb?

Btw I am happy I got only small amounts of data in any of bolt databases...

[ Parent | on HN ]

Re:Fly.io Postgres cluster down for 3 days, no word from the... by tptacek on Friday July 21, 2023 @08:11PM (#36821769)

This isn't a boltdb we designed. It's just containerd. I am probably not doing the outage justice, because "blitz and repopulate" is a time-honored strategy here.

[ Parent | on HN ]