If you have super high throughput, it would be worth temporarily (and very securely) caching transaction parameters to handle downtime.

While on paper it seems simple, it's worth investigating in detail how changing where/how payment details are transmitted and stored could change regulatory compliance requirements and liabilities of your business. It could be more time consuming and expensive than anticipated.

and hand-rolling means you've just swapped stripe as the failure point for your homebrew

How would you do this without suddenly becoming subject to PCI compliance?

One way would be to use a service like VeryGoodSecurity to capture payment details initially, then have it forward them to your processor later on.

Create your Stripe token client side, send it to your API, indicate to the user that the payment is processing.

Your backend stores the PCI-compliant Stripe token in a queue which a worker processes as and when it can - therefore allowing you to mitigate Stripe down time.

The issues then become one of UX if the payment fails.

If Stripe is down, you can't create a Stripe token. Iirc tokens also expire fairly quickly (at least - in my testing, that appears to be the case. Perhaps it's different for different types of tokens.)

Are Stripe's systems are isolated enough to where their token system is disjoint from the charge system? Do we know what uptime for their token system looks like?

That’s a fair point about creating Stipe tokens.

In my experience (processing several thousand payments with Stripe daily) when there are blips they do seem to be isolated to specific endpoints/entities.

This is a good idea. I've been working out a plan to move transaction processing to background processes to help with web throughput. I'd imagine I could solve for this problem at the same time.

I've thought about this, but I wonder what the UX is like.

You always show success? What sort of confirmation does the user get? If the card is declined, how do you notify them later? Would that notification confuse them?

So many things to think about.

"Thank you for placing order number x. Check your email for confirmation."

Email is somewhat immediate if the gateway was up, somewhat delayed if it was down. Regardless, it then offers order confirmation and shipping info, or it offers a card-declined-try-again flow.

> Losing a ton of revenue

How much would it have cost you to have never used Stripe?

There are alternative services. Some offering better conditions.

Care to share them? So far I've found that only Stripe provides such a high quality of service for the pricing, but I'm genuinely interested in alternative!

Braintree?

> Losing a ton of revenue and it is completely out of my hands.

That may be a bit exaggerated. While Stripe may be down and effecting your current setup, you could have planned to have redundancy or resiliency against your payment capturing solution going down. No technology never breaks.

Yeah, depends on your business, but for us Stripe is only necessary for new customers or for folks to update their billing information once in a blue moon. I definitely envy anyone getting multiple new customers per minute.

Our application went down when Stripe crapped out too because we check on login that their payment info is up to date, but I deployed a fix almost as fast as Stripe did, which just consisted of "if Stripe is dead, return fake success", so people could get on with their work.

Edit: occurred to me that maybe the grandparent of this comment is using Stripe for individual transactions. If so, may I suggest you use a payment processor that won't take 2.9% + 30 cents per transaction? Those are relatively high rates. Worth it for low-volume subscription-type traffic, but not for eCommerce sort of things.

Edit 2: regarding the previous edit, it's complex, and it depends. You do you.

Do you have any payment processors to suggest who don't take cuts that high?

You can negotiate with Stripe if you're at high enough volumes. It's likely that the "best" choice of payment processor is heavily dependent on the specific business in question. If you need agility and developer friendliness, Stripe is hard to beat. If you're trying to grind out every last percent of margin, you'll have to shop around and see what you can negotiate (and the offers you get will likely depend on the nature of your business, chargebacks/fraud, etc).

I have to admit I was thinking primarily of my company's use case, which is serving brick-and-mortar. This is a pretty different picture from card-not-present transactions, but if you're a low-risk business from the point of view of credit card processors, 2.9% is still at the high end. If you're brick-and-mortar, you can get rates as low as .25% sometimes.

Fattmerchant, Gravity Payments, and Worldpay are all great options for brick and mortar, and offer online payments too. Paypal is also cheaper than Stripe for US businesses.

As always, it depends, and it's complex. I probably was too confident in my above answer.

disclaimer: I work at Gravity Payments AMA.

Stripe is an aggregator, which means they collect all payments and distribute to their clientele. This is why merchant processors like Square and Stripe can often get their customers up and running more quickly. Lower underwriting requirements = less regulation on the merchant. The level of risk is higher so they have to charge higher rates to cover their losses of fraud.

Gravity Payments is an Independent Sales Organization (ISO) which means they underwrite each merchant and "approve" each merchant account with their backend processor. This equals less fraud and more flexible pricing.

We do offer integrations and also have an online product that can process ecomm transactions for developer usage.

> if Stripe is dead, return fake success

Just a thought, might want to make sure that cannot be exploited by blocking the Stripe API when someone logs into your app?

I would assume that the check is

1) made server side, so can't be blocked by the client

2) only made to prompt people to update their payment info if needed

Correct, the check is server-side. Nothing the client can do about it.

Except for DOS attacks to Stripe /s

That'd only work if the client/web page is doing the API calls, not if the backend was doing them, right?

Easier said than done. Outside of the costs and overhead required to implement a secondary payment solution in the rare case your primary solution goes down, often times payment providers require exclusivity agreements which prohibit this.

Never said it was easy, but if it's bad enough to make a post about how "painful" it is and "losing revenue" it should be a consideration? Yeah?

When P(payment provider outage) x (expected lost revenue in case of outage) > (costs of implementing and maintaining an independent alternative payment processor and automatic failover), then hoping for the best and writing a comment about pain in case of outage is still the best strategy.

Lets say it takes 2 engineers 3 months to build an another payment processor + failover, at $120k salary + 40k in benefits and taxes a year a piece.

stipes SLA was 99.980% uptime for the last 90 days

0.02% * revLoss > 160k * 2 * .25/year

OP's app would have to be earning 400mm a year. Not likely but possible.

400 million a year if all revenue comes from impulsive people that won't simply try again an hour later.

Most likely it's worth taking the revenue loss rather than building a redundant payment provider. That doesn't mean taking the revenue loss doesn't hurt.

Exactly. If it starts happening frequently, maybe it's worth looking into. But I think there's a middle ground between 'worthy of complaining on hacker news' and 'need to implement and support a redundant backup payment system.'

KATG listener since 2006. Awesome to see you on here.

I’m reading your reply like “Yup. Yeah. Totally. Wait, what?!”

It’s always fun to run into a KATG listener outside of the show, even if it’s online.

I’ll show this to Keith and Chemda :)

You’re right, you would never believe the amount of effort required to make an HN post. I’ve often not only implemented backup payment provider solutions instead, but also tertiary ones. In fact, in lieu of this comment I was 90% of the way to starting my own payment network.

I get that point, but I run a platform powered by Stripe Connect. Redundancy at that level would require the customers who sell their products through my platform to set up an additional account, go through additional KYC, etc - which is unrealistic. Alternatively, I could register my business as a PayFac, which costs a ton of money and depending on your network, also faces outages from time to time.

Sometimes things truly are out of your hands.

what's to exploit? If they're logging in, they have the credentials already.

I get that someone could maybe somehow avoid updating the stripe info, but that will fail the next time a charge goes through, so it's not as if there's a lot of fallout from it. Without even questioning why someone would go through the trouble in the first place.

Just curious, do you have a proposed solution?

The best I can think of would be to have a feature toggle that can be manually flipped by a developer and route transactions through PayPal when the toggle is flipped. This would solve the ability to collect payments for new customers, but there would have to be some sort of reconciliation/sync when Stripe comes back up to migrate the customers back to Stripe, otherwise you'll have a handful of customers in PayPal indefinitely.

Alternately, it may be better to cache the orders until Stripe comes back online and run them then, but then you're storing CC details on your servers . . .

You could find a way to do a soft failure. In the case of as payment gateway failure, take the order and follow it up manually once the gateway is back up. May not work for every business but an eComm business could just hold the fulfilment until payment is captured. A subscription service could let the subscription run on a short grace period and follow it up.

That could all be done programatically, or give them a call or email.

> you could have planned to have redundancy.

Not really. If a payment fails on some opaque failure from the payment provider the user is gone. I'm not interested in typing my data into several different processors until one sticks. I'm looking for your product somewhere else. Payments must work.

>That may be a bit exaggerated. It's really not though. As of time of writing, the customers have failed to sign up, and there's nothing to do about that on the fly, right now, today. Saying you "could have done X" doesn't mean that the problem isn't happening.

"Your house isn't on fire, you just haven't properly fireproofed it" isn't really helpful to anyone when their house is literally on fire.

> No technology never breaks

Not true. It just takes more effort to be more resilient. Totally possible. Think of telephone line.

If someone cuts the phone lines into a building, whether with malice aforethought or a badly aimed backhoe, the phones in that building will not serve their intended purpose.

I agree that degrees of resilience are a thing, and that different kinds of systems have different failure modes (each of which may deal with a different aspect of resiliency), but I am firmly convinced that no technology never breaks.

Degrees is a key word. Numbers matter. What was downtime of your telephone line over last 5 years? What was downtime of Wikipedia over last 5 years?

Well, Wikipedia's uptime in 2017 was 99.97%, according to [1]. That's over 2h30m, so if this is Stripe's first downtime of the year, they're still in the lead.

[1] https://wikimediafoundation.org/technology/

> What was downtime of your telephone line over last 5 years

Assuming they even have a 'telephone' landline, are they even measuring?

Plus, telephone is old tech. This is apples to oranges. You know what else hardly ever fails? The water supply to my house. But people have been building aqueducts since Babylon.

My phone line was down for 4 days because they accidentally disconnected it when hooking someone else up and it took them that long to fix.

Alas, even the mighty telephone line is not infallible.

This is where solutions like Spreedly and TokenX make a lot of sense. Once the payment method is stored in their vault, you can try (and retry!) payments on multiple gateways.

Genuinely curious — what happens when Spreedly or TokenX are down, though?

You would have the same sort of downtime as Stripe today.

But the tokenization wasn't failing for us in today's Stripe downtime -- just the API requests.