Re:A lot of complex “scalable” systems can be done with a si...
by inetknght on Tuesday December 31, 2019 @03:33AM
(#21919722)
Your problem sounds quite similar to a lot of FinTech interview questions for software engineers. They're solved problems but solutions aren't easy.
I have no doubt I could solve your problems. I honestly don't care to do so here though.
I imagine there's a software development/engineering team at your work or school you could ask for guidance though.
Re:A lot of complex “scalable” systems can be done with a si...
by hnews_account_1 on Tuesday December 31, 2019 @05:39PM
(#21925262)
> I have no doubt I could solve your problems. I honestly don't care to do so here though.
That's fair. I did not think it'd be such a difficult problem when I first set out to do it myself. But every single turn leading to a dead end kinda bummed me out. I'm going to finally resort to the database method of storing all the data in one query-able file and work off of that.
While I hope that each 250GB chunk has everything in order and I can separate it cleanly, I don't trust it. So I broke up all the files by stock in an embarrassingly parallelised code I wrote and got 500 separate files. However, the problem is, I'd want to process these 500 files parallely so each stock gets only so much memory (and hence, my original constraint remains). And for each stock, I want the data to be queried on timestamps, so I need a way to quickly say "I have data at 9am on Thursday, I want data from 4pm Wednesday to 8am Thursday to create a model".
I figured the best way to do this efficiently was to have the code create a massive database for all the stocks and query it efficiently in SQL. But I'm stuck there due to a lack of tools.