Ask HN: What DB to use for huge time series?

Re:Ask HN: What DB to use for huge time series? by siliconviking on Thursday September 25, 2014 @02:15PM (#8368657)

First question - do you even need a database right now? Have you for example considered using CSV files and simply loading those files into Pandas or R on demand?

I am currently working on a project analyzing massive amounts of options data and have found this approach to be both quite easy as well as flexible to work with... and as my project matures I may move select parts of it into a database.

[ Parent | on HN ]

Re:Ask HN: What DB to use for huge time series? by new_test on Thursday September 25, 2014 @02:32PM (#8368792)

>loading those files into Pandas or R

What is "massive" for you? I was under impression you can't use R or pandas for anything that doesn't fit into memory.

[ Parent | on HN ]

Re:Ask HN: What DB to use for huge time series? by siliconviking on Thursday September 25, 2014 @05:00PM (#8369717)

As for massive - something like daily options data for 3000 stocks, spanning a number of years, with information down to the tranche level (let's say 60 million rows if stored in a relational database fashion). In my case the analysis can be done on the stock level though, which means that only a 3000th of the dataset needs to be loaded into memory at any time.

[ Parent | on HN ]