The simple answer is that the reason the system failed the first time wasn't easy to fix in the short term. Either they need to acquire more hardware resources or redesign some system so that it scales better. Not everything can be fixed overnight. Take a look at reddit - that's a website that collapsed repeatedly for months and months and months. Sometimes it's not something you can just 'fix'.

I'd argue that the consequences of reddit not being available for a certain amount of time are absolutely not comparable with a financial services' firms infrastructure.

Mistakes, any mistakes, are very hard to excuse when people, or other financial entities happen to lose money.

Comparing it with a site like reddit is, frankly, a bit disengineous.

Consequences, no. Infrastructure failures, absolutely. What's being hosted isn't as important when it's broken and you're struggling to fix it. Sure, when it's a financial service's infrastructure, you're worried about other peoples money, but it's still something that's broken and may take time to fix.

The load testing you need to do for trading is considerably higher than any chat forum to go live .

Your customers are paying for speed and performance .

In a industry where users spend millions to be closer to the exchange if you are not provisioning for some multiple of your peak load you are going to get slaughtered in the market

I don't think that argument really holds up when Robinhood is the subject. I mean they still don't even have phone support or same day guarantees covering them responding to your email.

Their support is horrific. Will never use them again. M1 is delightful in comparison.

Robinhood is nothing like Reddit. It has a much bigger valuation, more resources, strict regulations to follow, and the responsibility of safely handling money for millions of people. This should be fixed within days even if it means redesigning the entire system.

The last blog post blamed overloaded DNS so it doesn't seem like a major architectural problem, which is why it's surprising that they're down again.

> This should be fixed within days even if it means redesigning the entire system.

Please tell me you do not manage a team of software developers.

The last 20 years I have been working in heavily regulated environments; in cock-ups like this were your Retail clients got royally screwed (for a second time in such a short period), on a repeat issue, the hammer will fall HARD on their heads. Anyone wanted to sell to minimize losses has been severely damaged.

There are many ways to resolve these issues FAST. The easy way is to throw money at this. Go tomorrow (literally) and bypass all procurement controls and go to a mega big provider and scale this asap. Any financial services with a half decent IT has done the paper exercise to this scenario (at least for the purposes of BCP/DRP). The slow/better/mature way may be too slow, especially with the current market conditions.

The RH folks will definitely get a visit from SEC, their external auditor, and their external auditor will get a visit from SEC.(their auditor will be in the deepest of shits)(how come they failed to spot such a going concern issue?)(what the hell were they looking for on their audits?)(did they only send juniors over there?)

I feel sorry for the retail traders that got knocked down. I think that anyone locked in buying at 27-28k (US30) should wait 6 months to breakeven and after the US elections (irrespective of the winner) there will probably be a rally.

This is why you would use IB for a retail brokerage account versus the "cooler app" if you care about execution and uptime.

They aren't that fantastic either. They had issues today as well, I couldn't exit out of spreads, so I had to liquidate everything and that was failing too. It was a hot mess. Edit: I didn't actually want to liquidate all my holdings but it was better than staying in and risking everything else.

"redesigning the entire system" was an exaggeration. They have a serious problem if that's really what they need to do.

Rolling out fixes fast, even if they require intensive changes, is completely reasonable and expected in many industries though. They should have the talent and procedures to get it done. This isn't some startup web app, it's a multi-billion dollar broker managing millions in client funds. It's bordering on incompetent to have 3 outages in 2 weeks.

EDIT: What exactly is everyone disagreeing with?

> Rolling out fixes fast, even if they require intensive changes, is completely reasonable and expected in many industries though.

This is a financial trading platform. Do you understand the risks of potentially introducing a different bug?

While I don't agree with OP about replacing an entire system overnight. I do remember a friend who worked on a trading floor, and when they had bugs, their manager would say "the traders are taking a 30 minute lunch, you have that long to fix it" with the clear implication that they'd be fired if they didn't.

So I'm not sure if I know that the state of the art for trading platforms is as rock solid as everyone is implying, and Robinhood seems to be way far off from whatever gold standards there are, see infinite leverage bug. So I don't think it's crazy that they move quickly to fix it.

I'd never be able to stomach the pressure, and I wouldn't wish it on others, but it doesn't seem crazy.

I understand the risks need to be weighed against restoring service for their users who might be losing money and avoiding regulatory fines.

Changing a single line can introduce a different bug. Use proper QA and testing to catch as many as possible as with any development.

My emphasis is on getting things fixed quickly. They need to do whatever it takes to get systems online asap. Not sure what's so controversial about that.

Because move fast and break things doesn't work in finance.

Nobody said that. Things are already broken. Move fast and fix things is what they need to do.

Well to be honest we don't know how badly things are broken only they do. But the concept of moving fast while doing things perfectly is the holy grail of software development and not as easily achieved as you seem to make it appear.

I only said it needed to be done fast, regardless of how much work. Of course it's not easy.

I'm surprised by all the misinterpretation in this thread. Seems like it reflects the laid-back West Coast/SV attitude that isn't a good fit for high pressure time-sensitive work in other industries.

You seem to think (or at least imply) that hard things can be done fast if only you work hard enough at it, that it's just a matter of trying. This is just not true. Not sure how else your posts should be interpreted. I've spent days with a team (yes a competent team) just tracking down a bug, let alone fix it (although usually when it's tracked down it's relatively quick to fix). If this issue involves multiple systems in a highly complex environment, then it very well could take a while to address fully, no matter how hard they work at it.

Because that's usually the case. Crunch time, disaster recovery, and emergency fixes are common in every sector from video games to aerospace. If you can't fix then switch to a secondary, or rebuild from backup, or throttle users, or process manually, or do anything other than be completely down.

RH wasn't prepared with any contingency. They should have a resolution for their users - even if they can't find or fix the original cause. That's the failure I'm talking about.

See the 2 other users in this thread that describe similar high-pressure situations.

Depends on what the issue is. Could be something that can be quickly fixed or not. Although going by the lack of a resolution I’m assuming it’s not.

Reality is, without knowing more about what’s causing this it’s impossible for either of us to say. If there is indeed some fundamental bottleneck that was previously not known, then I certainly won’t be surprised if it takes a while to sort out.

Now you can say they should’ve load tested, capacity planned etc etc. But we are where we are. Still can’t go back in time to turn this into a quickly fixable problem if it’s currently not.

Edit: also pretty disappointed that we don’t know more about the root cause. As an user I’d want to know what the issue was and what they are planning to do to about it to evaluate if I should trust them going forward.

> needed to be done fast, regardless of how much work

This is the part you don't understand. There is a difference between digging ten one-foot deep holes vs one ten-foot deep hole. People need time to plan how to coordinate and then get on the same page so that everyone can work at their own pace. That is the part that is not parallelizable and is the rate-determining step.

The context is all lost here. Plenty of other companies and industries have emergency action and disaster recovery. People don't work at their own pace, they work to the deadline with solid procedures. They can fix and replace entire components to restore service ASAP since because that's the priority.

If this sounds unfamiliar or onerous then it's because you and others might have never experienced teams that do this. Robinhood is clearly lacking this experience and disaster planning.

> strict regulations to follow, and the responsibility of safely handling money for millions of people. This should be fixed within days even if it means redesigning the entire system

Somehow, in my mind, the responsibility of safely handling money of millions and redesigning the entire system don't really mix. Ditto for strict regulations and redesigning the entire system.

I know it's not the same regulations but I'm pretty sure the next FDA auditor would have a heart attack if we slipped "rewrote the whole product" into one of our patch notes at work.

Also, Reddit can afford to queue and buffer like crazy.

Nearly every write-action on Reddit appears to hit a queue. If your vote or comment doesn't go up for 30 seconds, that's not a problem.

If your trade takes 30 seconds, that could be a huge deal.

Being in a regulated environment with lots of peoples money should make it harder and take longer to redesign and implement anything. I don’t see a complete redesign of a non-trivial part of a trading platform happening in a week.

That’s before outages, priority now is to restore service ASAP, whatever it takes.

Your literal quote was "This should be fixed within days even if it means redesigning the entire system". And my position is, being a regulated financial service, it's probably not reasonable to redesign (and build, test, deploy etc) a system in days.

If they KNEW it was broken and were working on a fix, then they should of shut down the service. I'm sorry but this is simply unacceptable. I don't use Robinhood, but when you are handling very time sensitive contracts that can make or lose people a lot of money in a very short period of time if the service is not stable it should not of been in operation until it was fixed. I can't believe HN is making excuses for this.

If they do that, then they're definitely harming any customers who have a position they need to get out of, as opposed to only potentially harming them.

That's exactly what is happening right now anyways. Except in the past week many new positions were opened on a broken platform.

If this was Wells Fargo we'd see the pitchforks. But it's a SV company so all is good.

I mean, the investment community knows, and even on other sites there was a lot of encouragement to get -out- before the next disaster struck.

IMO anyone who opened a position on RH in the last week should not be shocked that the same thing happened again in this high volume crapstorm.

If you believe that the platform is so broken that it shouldn't be used (and I'm not arguing against that, btw) then a more reasonable solution would be to only allow position-reducing trades. That would also reduce overall trading volume on their platform.

Not that I expect them to do any of this, just pointing out that 'shut everything down' is not necessarily the best approach.

Cloud computing resources are nearly limitless.

Provision the resources and take the financial hit in costs.

If your architecture wasn't able to scale horizontally because it was poorly designed, heads should roll. This isn't some social network - these are financial platforms where literally the individual customer is financially dependent.

Heads. Should. Roll. Tell us the post-mortems, and then tell us who got the boot. Completely unacceptable.

The only company in the world who can claim to have made a perfectly horizontal system at all levels of macro and micro is amazon, and even they would be lying to do so.

A system like robinhood can have hundreds of moving parts, if 99 of them are horizontally scalable but 1 is not, eventually that 1 piece will become the bottle neck, and the fact it hadn't been made horizontally scalable yet is more likely to be a testament to how much work would have to go into doing so.

> Heads. Should. Roll. Tell us the post-mortems, and then tell us who got the boot. Completely unacceptable.

This is an unfortunate viewpoint. How quickly did we forget that robinhood is literally providing a service that no other company was able to do before. You want 100% uptime, you won't find it in an online service, let alone an online fee free service, you'll find it on the floor of the exchange.

All brokers were able to do it, they just didn't because commissions were easy profit and there was no competition.

$0 commissions aren't a big deal either. If you just buy and hold, it makes no difference. If you trade actively then a real broker with better tech and order management is worth way more than the fees.

> The only company in the world who can claim to have made a perfectly horizontal system at all levels of macro and micro is amazon, and even they would be lying to do so.

Haha, true. Call up you AWS account representative and you may find that certain service limits can’t be increased for love or money.

> a service that no other company was ~~able~~ willing to do before

ftfy

If you’re limited by database throughput and can’t scale further, throwing more hardware into the mix doesn’t help.

If you horizontally scale all the way, you just take on different issues like CAP theorem trade-offs. Financial book-keeping is a hard thing to scale and all it takes is a single component that cannot, like the transactional bedrock of your system.

Just use Google Spanner

I don't think getting rid of the people who know the system best will help anything. Some new guy who comes in with no clue of what happened or any of the codebase is not going to do a better job. It's just going to reduce long term productivity and increase risk.

I get the impression that the people running Robinhood actually don't understand how difficult it is to implement what they want to offer, both on the reliability side (outages) and on the logical side (infinite margin).

This is pretty interesting for capacity planning. I worked at a minor ecommerce company when Black Friday was still a big deal. In the month leading up to it, we went into code freeze and made sure all our systems can handle added load.

What Robinhood is facing is like that, but with no notice. It means they ~always need to be ready for a 4x(?) increase in load.

This is one of the problems that "the cloud" is supposed to solve. No need to keep infra up for that unicorn event, just to ensure that you can scale properly when it does happen.

Until you hit some kind of aggregation server that it's not trivial to scale out properly unless you take a completely different approach or that you may scale vertically up (but eventually you will hit the ceiling).

What sets financial services apart when they fail is the extreme legibility of the losses. One can argue that if say Reddit is down, they have losses, but measuring the impact in terms of dollars lost is difficult.

It's a lot easier with something like Robinhood. So, ethics aside, the level of measurable risk they have now is existential. Lawsuits will flow.

I would imagine the volume differential scale is massive here, they probably are scaled at 1X peak, so they are capable of handling ~3X peak.

They are probably at ~10X peak traffic at times.

Just are not prepped for a hardcore crash.

Looks like they had a bug w/ leap years and load.

The issue a few days ago was not related to the leap year. That was only a theory at the time. They have since claimed it was a DNS issue.

I would not be surprised if many trading places have been told to shut down temporarily.

[1] https://www.ccn.com/dow-crashes-so-hard-might-close-stock-ma...

[2] https://www.businessinsider.de/international/stock-market-ne...

[3] https://www.bloomberg.com/news/articles/2020-03-08/rout-in-u...

The market makers and exchanges stop executing orders because a series of circuit-breakers fired (set by the regulating organizations).

Robinhood and other brokers don't execute any trades, they just forward orders from users to their exchange partners and show up to date quotes. They have a much easier load to bear than actual exchanges.

That’s about a set of specific, known “circuit-breakers”. They shut down trading at the actual exchange, not “ask trading places“ to do so.