jjHey

Very good remark! Storm by default implements at-least-once semantics. If a bolt process executed an event and failed right away without acknowledging, an execution would get replayed (so you would buy/sell twice). However, there's a remedy! By sacrifacing latency you can have exactly-once-semantics, see Trident, Summingbird.

No. I only had boxes with SATA drives. I don't see how you could get below 1ms latency with spinning disks. You would have to use in-memory queue to get this sort of latency. Secondly, I had Storm consuming off Kafka every millisecond, so in the worst case scenario a message would have to wait one millisecond in the queue anyway. Here's a great article about throughput/latency of Kafka https://engineering.linkedin.com/kafka/benchmarking-apache-k....

Thank you for the comment. I agree that having sound data is essential to building any trading algorithm. Probably that's why they are so expensive. I won't complain about HistData.com, it's a free service, they only charge a small fee for a convenient ftp access to their site. At the moment, that's the most you can get for free with a millisecond resolution. Despite it's drawbacks, works well for backtesting, IMHO.

A great alternative is finance.yahoo.com. It is reasonably rate limited, see http://stackoverflow.com/questions/9346582/what-is-the-query.... You won't be able to do any high-frequency trading anyway without dedicated hardware in a dedicated data center.

That is a very good point. An initial idea of Wolf was to create a platform that could trade virtually anything (Craigslist, Amazon, Ebay, Stock market, Forex, etc) as long as there's a trading API, and could make use of data from just about anywhere. Not just ticks. Imagine you can write a trading rule like this: "if Mr. X invested $ in company A and Ms. Y recommends buying stocks of company B or an epidemy spreads in country C or product D received great consumer reviews then trade E for F". That would require a lot of data: news articles, financial data, customer reviews, geopolitical events, etc. One of the Insight data engineer found some cool correlations between geopolitical events from GDELT and Forex. Wolf was created end-to-end in 6 weeks so... I focused on Forex. P.S. One year of compressed Forex ticks from HistData.com is about 100GB.

A pilot installation of Wolf was made on 10 ec2 boxes, mainly m1.medium and m1.large if I remember correctly. 3 boxes for C*, 3 boxes for Hadoop, 1 for Kafka, 1 for Storm, 2 for everything else. You might be able to squeeze it in less machines as Amazon is upgrading specs all the time. Deployment scripts are very simple at the moment, just a bunch of shell commands. That might be upgraded in the future to some magic automation. Nevertheless, scripts were tested on Ubuntu 12.04, but are not specific to Ubuntu, other then the package manager (apt-get). Probably if you sed s/apt-get/yum the scripts, they will work on CentOS/Fedora. If you have a really strong machine you might even be able to squeeze everything in one box. I wouldn't recommend it though, keeping the state of all the clusters in one Zookeeper is a very risky business, plus reliability issues. Unfortunately, it's not so simple to get Wolf from nothingness to a working/trading system. To actually trade you will also need a data feed, and a brokerage account.