News for hackers. Stuff that matters.
Re:Robinhood goes down again, during another historic tradin... by lm28469 on Monday March 09, 2020 @10:26AM (#22525583)
This is a good reminder that all these new mobile(brokers|banks|&c.) are not as good as the historical systems that have been stress tested for decades. All is good when everything works as expected but as soon as the slightest issue arise you're in for a ride.
Re:Robinhood goes down again, during another historic tradin... by dx87 on Monday March 09, 2020 @10:37AM (#22525682)
That's what I think of every time I see a post about some new VC backed startup that says they are going to disrupt an industry by creating a lean product that cuts out all of the unnecessary cruft and legacy systems. People need to learn that there's usually a reason something is done a certain way, and having barrels of VC money doesn't make you an expert in a field.
Re:Robinhood goes down again, during another historic tradin... by greenhatman on Monday March 09, 2020 @11:06AM (#22525963)
Sometimes the reason it's done that way, is because they were in a rush in the first iteration, and never got around to refactoring.

Loads of VC money means you can take more time up front to architect things better. Maybe have more experiences engineers and architects on the team from the start.

Re:Robinhood goes down again, during another historic tradin... by papito on Monday March 09, 2020 @01:15PM (#22527196)
Not really. VCs expect 10x returns. Yesterday. They really don't care if it's the best system ever designed. It just has to work until a liquidity event.
Re:Robinhood goes down again, during another historic tradin... by elamje on Monday March 09, 2020 @10:29PM (#22532216)
I understand your point, but I think VC backed companies are likely in a more hyper growth mindset than banks and conservative industries were when they were making these systems in the 80’s.

As much flak as I’ll get for saying this, why Python? It seems like Robinhood went all in on an ecosystem that is easy to get an MVP off of the ground with, but certainly not as fast and reliable as you would expect a system moving billions everyday. Language typically doesn’t matter for end users, but man, there are at least 10 solid options to build a critical trading system in that are faster and safer than Python.

Re:Robinhood goes down again, during another historic tradin... by CameronNemo on Tuesday March 10, 2020 @12:43AM (#22532779)
Is it really 100% python? I can't find any indication of what they use besides the python helper library for Kafka. Seems like a message queue would be conducive to partial rewriting of the system.
Re:Robinhood goes down again, during another historic tradin... by naskwo on Monday March 09, 2020 @10:38AM (#22525696)
Re:Robinhood goes down again, during another historic tradin... by brain5ide on Monday March 09, 2020 @10:49AM (#22525794)
Pasting this has recently become a meme. Not sure it applies here, as it's not about removing stuff from the same instance, but about a competitive lean product?

Innovators dilemma?

Re:Robinhood goes down again, during another historic tradin... by vikramkr on Monday March 09, 2020 @10:58AM (#22525886)
It's about removing legacy systems and pretty directly about reforming xyz industry with the new tech, so I think the principle of reform requiring knowing why something you want to get rid be of is there in the first place is directly related to the topic
Re:Robinhood goes down again, during another historic tradin... by salawat on Monday March 09, 2020 @11:43AM (#22526300)
A quirk of information propagation. If it's there, it was put there for a reason. If you don't understand the reason, you're at risk of having to rediscover it the hard way; something we generally wish to avoid at all costs.

Which is somewhat ironic considering the attitude around things like Trade Secrets leading to a plethora of Chesterton's Fences being basically guaranteed to litter the economic landscape.

Re:Robinhood goes down again, during another historic tradin... by strbean on Monday March 09, 2020 @04:45PM (#22529582)
I think the real irony is that much of the time fences exist because of the quirks (fences) of some other system you are stuck with. Sometimes it feels like it is fences all the way down. Other times, you work your way down 5 layers of dependencies and discover that the whole pile of ugliness is because the core system is designed around physically mailing boxes full of reel-to-reel tapes. When that is the case, there is potential for a cascade of "lean modern apps" to de-cruft an industry.
Re:Robinhood goes down again, during another historic tradin... by robjan on Monday March 09, 2020 @10:58AM (#22525885)
Ultimately, RH is backed by a legacy system somewhere. I believe they custom built their OMS (order management system) which was the previous bottleneck but somewhere between RH and the other side of the order is a legacy system that they can't disrupt.
Re:Robinhood goes down again, during another historic tradin... by dannyw on Monday March 09, 2020 @11:23AM (#22526098)
To claim that RH would outscale NYSE, NASDAQ, and market makers like Citadel and Renaissance Technologies is absurd.

It's like saying Airbnb is taking down root level DNS servers.

Re:Robinhood goes down again, during another historic tradin... by dehrmann on Monday March 09, 2020 @11:49AM (#22526359)
> Renaissance Technologies

I didn't realize they were a big player in this game. Citadel definitely is, though.

Re:Robinhood goes down again, during another historic tradin... by manigandham on Monday March 09, 2020 @11:28AM (#22526153)
RH doesn't fill anything, they only forward orders. All of the exchanges and other brokers are up and running just fine.

This is strictly an RH problem. Their last blog post said it was a DNS problem.

Re:Robinhood goes down again, during another historic tradin... by robjan on Monday March 09, 2020 @12:08PM (#22526511)
Fully aware of that. RH don't even directly connect to the exchange, they use a third party execution broker.

Since all securities are affected, it probably actually is their OMS which is malfunctioning.

Re:Robinhood goes down again, during another historic tradin... by yumraj on Monday March 09, 2020 @11:22AM (#22526094)
If that were true, why are other equity trading systems, and by that I mean every other trading system, doing fine?

The problem is RH itself.

Re:Robinhood goes down again, during another historic tradin... by jeffshek on Monday March 09, 2020 @10:54AM (#22525842)
This might be slightly unfair to Robinhood ... the time we saw volatility similar to this was in 2008. Most of the major banks had technical difficulties that day (despite being around for decades of years).

Technology and scaling are tough. Reddit, Twitter and CloudFlare have gone down in the last 12 months, despite being constantly stress-tested.

Re:Robinhood goes down again, during another historic tradin... by CaptainZapp on Monday March 09, 2020 @11:02AM (#22525928)
Comparing social media sites with a financial services firm is not really a valid comaprison.

Especially not when this happens for the third time in a row.

Re:Robinhood goes down again, during another historic tradin... by behohippy on Monday March 09, 2020 @10:35AM (#22525664)
My direct trading account at my (very) old bank is also saying it's having technical difficulties today. It sounds like display issues, not trade execution issues though.
Re:Robinhood goes down again, during another historic tradin... by geerlingguy on Monday March 09, 2020 @10:42AM (#22525728)
I know Fidelity changed their default dashboards to not load data-heavy views for the past week or two. Usually you get a nice dashboard page with some charts and personalized info, but right now you get a few generic (non-customized) market stats and a bunch of generic links.

Still useful enough, and if you want, you can get to your own dashboard... but things like this (inconsequential as they seem) can be the difference between going down or not.

Each non-personalized element on the page that's removed saves at least one DB query (and in a fancy app like Robinhood's case, maybe a half dozen).

Re:Robinhood goes down again, during another historic tradin... by dirtydroog on Monday March 09, 2020 @10:37AM (#22525686)
Well, market circuit-breakers are being triggered all over the world. I doubt anyone would be able to trade even if they could log in.