Ask HN: All of you working with Big Data, what is your Data?

Re:Ask HN: All of you working with Big Data, what is your Data? by noname123 on Saturday January 24, 2015 @09:43AM (#8939679)

I run backtesting for options trading as a hobby and storing EOD tick data, querying it and extracting it is a pain. I dl my source data currently from a retail historical data provider, then store it in MongoDB in AWS. What would you recommend tech stackwise to do backtesting on time seriea data?

[ Parent | on HN ]

Re:Ask HN: All of you working with Big Data, what is your Data? by dmichulke on Monday January 26, 2015 @03:26AM (#8946017)

I also first used MongoDB but I switched to cassandra and I find it much better.

MongoDB was bad for this case because it devours memory and it lacks a primary key (so saving a tick that's already creates a second tick).

cassandra is better (it has a PK, although it's a bit weird because of its distributed-first attitude) but in the long run I think postgres would be even better (because it's space efficient).

Apart from that I use some Java libraries and clojure/incanter and program the rest around it myself.

[ Parent | on HN ]

Re:Ask HN: All of you working with Big Data, what is your Data? by jaredbroad on Saturday January 24, 2015 @10:32AM (#8939813)

QuantConnect.com has tick data for every stock in the US universe back to 1998 -- its roughly 4TB of data.

[ Parent | on HN ]

Re:Ask HN: All of you working with Big Data, what is your Data? by remyp on Saturday January 24, 2015 @09:55AM (#8939711)

They don't have options data, but you might check out QuantConnect or Quantopian if you're interested in stocks and FX.

[ Parent | on HN ]

Re:Ask HN: All of you working with Big Data, what is your Data? by leoplct on Saturday January 24, 2015 @09:56AM (#8939717)

Have you tried quantopian.com ?

[ Parent | on HN ]