News for hackers. Stuff that matters.
Re:Ask HN: All of you working with Big Data, what is your Data? by dmichulke on Saturday January 24, 2015 @09:33AM (#8939659)
Financial data (tick to EOD), network traffic data (TCP packet level sends / receives) and farm data (sensor + farm ERP data)

All of them are basically time series with some master data, none of them is more than a few dozen GB

So in any case, I think time series data is worth a look.

Re:Ask HN: All of you working with Big Data, what is your Data? by wglb on Saturday January 24, 2015 @10:26AM (#8939789)
You can get vast amounts of stock option data from http://www.orats.com/.

I don't think of the options stream as big data. It is more of a "fast" data problem, where at the open you need to handle 100,000 ticks per second, doing a implied volatility calculation on each tick. It is pretty demanding, when, for example, you consider that when the price of IBM moves by one penny, you get 400 options quote right then. You end up with a billion quotes in a typical day, if memory serves correctly.

I didn't think of this as Big Data, somehow. It seems to me that big data is more complex than that. You most often just process the data, say for backtesting, in a strict series, from the start to the end, or for an interval. I think of Big Data having more structure.

Re:Ask HN: All of you working with Big Data, what is your Data? by noname123 on Saturday January 24, 2015 @09:43AM (#8939679)
I run backtesting for options trading as a hobby and storing EOD tick data, querying it and extracting it is a pain. I dl my source data currently from a retail historical data provider, then store it in MongoDB in AWS. What would you recommend tech stackwise to do backtesting on time seriea data?
Re:Ask HN: All of you working with Big Data, what is your Data? by dmichulke on Monday January 26, 2015 @03:26AM (#8946017)
I also first used MongoDB but I switched to cassandra and I find it much better.

MongoDB was bad for this case because it devours memory and it lacks a primary key (so saving a tick that's already creates a second tick).

cassandra is better (it has a PK, although it's a bit weird because of its distributed-first attitude) but in the long run I think postgres would be even better (because it's space efficient).

Apart from that I use some Java libraries and clojure/incanter and program the rest around it myself.

Re:Ask HN: All of you working with Big Data, what is your Data? by jaredbroad on Saturday January 24, 2015 @10:32AM (#8939813)
QuantConnect.com has tick data for every stock in the US universe back to 1998 -- its roughly 4TB of data.
Re:Ask HN: All of you working with Big Data, what is your Data? by remyp on Saturday January 24, 2015 @09:55AM (#8939711)
They don't have options data, but you might check out QuantConnect or Quantopian if you're interested in stocks and FX.
Re:Ask HN: All of you working with Big Data, what is your Data? by leoplct on Saturday January 24, 2015 @09:56AM (#8939717)
Have you tried quantopian.com ?