Re:Stripe's API was down
by edwinwee on Wednesday July 10, 2019 @01:11PM
(#20403990)
We're back up as of 17:02 UTC: https://twitter.com/stripestatus/status/1149002362691833856
[ Parent
| on HN ]
Stripe CEO here.
We're very sorry about this. We work hard to maintain extreme reliability in our infrastructure, with a lot of redundancy at different levels. This morning, our API was heavily degraded (though not totally down) for 24 minutes.
We'll be conducting a thorough investigation and root-cause analysis.
Re:Stripe's API was down
by JaimeThompson on Wednesday July 10, 2019 @02:26PM
(#20404815)
Will the results of the investigation and analysis be publicly available?
This. Reading well-written post-mortems of outages for big and complex services like Stripe is just pure joy to me and feels very educational too. I remember reading Gitlab post-mortems earlier this year, and it felt really fresh, given how honest and open they were in those.
They're really helpful in preventing our own outages. Although it's ironic how many of them boil down to having things so well automated that one mistyped command can take down an entire environment with extreme efficiency.
Re:Stripe's API was down
by calhoun137 on Wednesday July 10, 2019 @06:12PM
(#20407141)
This is causing a big problem for my business right now, but I am not mad at Stripe because you earned that level of credibility and respect in my opinion. I understand these things happen and am glad to know a team as excellent as Stripe's is on the job.
Re:Stripe's API was down
by simonebrunozzi on Wednesday July 10, 2019 @07:12PM
(#20407543)
Patrick, I think that it would be really nice to share the technical details of the post-mortem, once the dust has settled.
Many folks don't have the privilege to run a massive scale operation like Stripe, and lots of people can learn from it.
Re:Stripe's API was down
by perfect_wave on Wednesday July 10, 2019 @02:17PM
(#20404722)
I also look forward to reading the postmortem. Stripe puts out a lot of high quality blog posts.
Re:Stripe's API was down
by jamestimmins on Wednesday July 10, 2019 @02:28PM
(#20404849)
The Stripe outage that occurred bc of a deleted DB index was particularly interesting.
Someone deleted an index before the replacement was live due to a process error. This had cascading effects across the system and caused a large % of API requests to timeout. It's such a pedestrian problem but had an enormous impact.
Aaaand it'd down again as of 19:19 UTC.
Wow, Stripe is having a really shitty day.
Re:Stripe's API was down
by mattigames on Wednesday July 10, 2019 @07:09PM
(#20407522)
Last tweet (10:58 PM UTC) says recovered already https://twitter.com/stripestatus/status/1149090037054623744
Re:Stripe's API was down
by omnimkar69 on Wednesday July 10, 2019 @01:34PM
(#20404282)
FROM 16:36 - 17:02 STRIPE'S system saw elevated error rates and response tims with the API. THEY HAVE NOW RECOVERED And are continuing to monitor as per their tweets on twitter
Re:Stripe's API was down
by kennethfriedman on Wednesday July 10, 2019 @01:12PM
(#20404003)
that was fast!