News for hackers. Stuff that matters.
Re:Robinhood reports outage as markets rebound by halotrope on Monday March 02, 2020 @05:43PM (#22469862)
Apparently leap day is the cause: https://mobile.twitter.com/jtech63/status/123460004578739404...

Also rumor has it they “lost user data and the backups are not working as expected”

Edit: This is the tweet. Not sure if it is legit: https://mobile.twitter.com/classrobinhood/status/12345774264...

Re:Robinhood reports outage as markets rebound by enlyth on Monday March 02, 2020 @06:46PM (#22470372)
It looks like the guy on Twitter doesn't understand that the API is down so you get 503 and a CORS error cause the pre-flight request failed and doesn't have the right headers. It will fail for any date.

The question is really does the front-end normally try to query the next day and what's going on in their order execution system on the backend, which we don't know because this source is just speculation based on console logs in the browser.

Re:Robinhood reports outage as markets rebound by ecnahc515 on Monday March 02, 2020 @07:18PM (#22470586)
My favorite part is this comment: https://twitter.com/holman/status/1234620062398464002 https://www.reddit.com/r/RobinHood/comments/48mep4/robinhood...

Additionally, if you’re a conspiracy theorist — and who isn't, in this wild 2020 ride — Robinhood was down exactly four years ago today, too, lol

Re:Robinhood reports outage as markets rebound by nodesocket on Monday March 02, 2020 @07:29PM (#22470631)
> Same here CALL ROBINHOOD ASAP. I spoke with them myself and they are aware. IF YOU NEED TO MAKE TRADES THEY CAN DO IT FOR YOU!

I didn't even know they had a phone number you could call. This should have been emailed out after the first hour of the outage. Or, how about you have a better system down page then just a blank website? If API health checks fail, serve a static site from S3 (route53 can do this). The ops failure of a company that can afford to pay quality DevOps engineers $200-300k a year is beyond belief.

Simple as:

"We are experiencing a technical outage... <corporate blah blah blah here>. If you need to make trades, please do so via phone at 1-800-SHORT-RH"

Re:Robinhood reports outage as markets rebound by EpicEng on Monday March 02, 2020 @09:18PM (#22471266)
>"We are experiencing a technical outage... <corporate blah blah blah here>. If you need to make trades, please do so via phone at 1-800-SHORT-RH"

"You're call will be taken in the order it was rexeived. Current wait time: 1,732 minutes."

Re:Robinhood reports outage as markets rebound by manigandham on Monday March 02, 2020 @07:04PM (#22470490)
This person is probing an API which is already down so of course there are errors.

Robinhood would be using a proper date library anyway to handle timezones. The system was fine on the 29th, 1st and even this morning pre-market. Seems more like a capacity issue that cascaded into failure.

Re:Robinhood reports outage as markets rebound by chance_state on Monday March 02, 2020 @11:38PM (#22471819)
The chances that it was also down exactly 4 years ago today (it was) are astronomical though.
Re:Robinhood reports outage as markets rebound by UncleMeat on Tuesday March 03, 2020 @11:50AM (#22476188)
The 2nd was a Wednesday four years ago. That wouldn’t make any sense for a leap year error. Why wouldn’t it have gone down on the Monday two days prior?
Re:Robinhood reports outage as markets rebound by nodesocket on Monday March 02, 2020 @06:40PM (#22470325)
As somebody who does DevOps consulting I was baffled how they could be down all day if it was a pure operations issue. That tweet is just speculation though. Hopefully more facts and details come out.

My bet is still on it being a database issue. If they indeed are using AWS RDS, I wouldn’t be surprised if they have half of AWS support engineers working on it. Once again, if I had to bet, I say they rolled their own "half-baked" database solution instead of using a AWS native offering.

Re:Robinhood reports outage as markets rebound by yushuf on Monday March 02, 2020 @06:58PM (#22470446)
I recently interviewed there and they said they're still using Postgres master-slave setup most likely on some cloud provider. Could be catastrophic failure with multiple shards going down at the same time. This might also explain why they mentioned their "backups" not working
Re:Robinhood reports outage as markets rebound by nodesocket on Monday March 02, 2020 @07:02PM (#22470469)
I believe they are on AWS, so maybe RDS PostgreSQL. If so they should be getting unlimited support from top tier AWS engineers.
Re:Robinhood reports outage as markets rebound by opsunit on Monday March 02, 2020 @07:30PM (#22470638)
If that is true and they're not on the ball they have another hard deadline fast approaching: https://aws.amazon.com/blogs/database/amazon-rds-customers-u...

(yes: the URI says February 5th whilst the article says March 5th!)

Re:Robinhood reports outage as markets rebound by aeyes on Wednesday March 04, 2020 @10:40PM (#22490750)
Why would they get unlimited support from "top tier" AWS engineers?

A company like Robinhood doesn't strike me as a strong candidate for contracting AWS Enterprise support. For the price you get almost nothing out of it.

Re:Robinhood reports outage as markets rebound by vasco on Thursday March 05, 2020 @04:00PM (#22498184)
Usually Enterprise comes with other benefits that, for large enough bills, offsets the extra cost support cost.
Re:Robinhood reports outage as markets rebound by pfarnsworth on Monday March 02, 2020 @06:23PM (#22470182)
This looks like a terrible source. I would wait until they issue a statement.
Re:Robinhood reports outage as markets rebound by halotrope on Monday March 02, 2020 @06:27PM (#22470221)
Very true! Unfortunately RH went completely dark. I sincerely hope they fix the issue and provide a post mortem.
Re:Robinhood reports outage as markets rebound by vsareto on Monday March 02, 2020 @06:26PM (#22470212)
If this is true and nothing happens to this company, everyone should feel free to use bugs to cover up any and all IT negligence.
Re:Robinhood reports outage as markets rebound by halotrope on Monday March 02, 2020 @06:33PM (#22470268)
I doubt it will be possible to just shake it off as there is likely millions of dollars lost for customers. Not in some abstract way like when github goes down but literally because people cannot close out positions and options expire. If it is really just a leap year bug It should be gross negligence. Not sure if their TOS will hold then.