News for hackers. Stuff that matters.
Re:Show HN: I'm building an open-source, high-frequency trad... by white_devil on Monday April 15, 2013 @08:12AM (#5550993)
Are you trying to get hired by the financial industry, or is there some other reason for doing this?

As you may well be aware, HFT is a scourge on the world's economy, and it's a game only the biggest and best-connected players benefit from.

Re:Show HN: I'm building an open-source, high-frequency trad... by zygomega on Monday April 15, 2013 @08:28AM (#5551051)
No, I'm already in the finance industry. There are two main reasons I'm doing this: - I think the finance industry is very closed when it comes to intellectual property development, and an open source approach can be seriously competitive. An open approach may well be the future when it comes to being 'connected' - HFT is an interesting multi-disciplinary problem and the shear breadth of expertise required - modern chip design, low-latency software/hardware interaction, lock-free concurrency, fault-tolerant system design, adaptive learning algorithms, k-means clustering - means I'm learning heaps every day.

I just don't agree that HFT is a scourge. It's an ecological shift (neither good or evil) and longer-horizon investors need to evolve.

Re:Show HN: I'm building an open-source, high-frequency trad... by jwilliams on Monday April 15, 2013 @08:44AM (#5551132)
Trading (as in HFT) isn't the same as investing. While the mechanics can sometimes be similar, a trader is different from an investor.

What HFT does is segment the market. It's probably best described as a form of arbitrage. Arbitrage is necessary, but in a good market hopefully it is pushed to some form of equilibrium.

HFT has an impact on the trading market. However, that doesn't automatically extend to investment.

Re:Show HN: I'm building an open-source, high-frequency trad... by astrodust on Monday April 15, 2013 @10:49AM (#5551742)
HFT is like a million tiny trolls underneath the bridge trying to extract their tax.
Re:Show HN: I'm building an open-source, high-frequency trad... by jarek on Monday April 15, 2013 @12:47PM (#5552479)
... replacing one big troll on the bridge that used to extract more than the million tiny ones' total...
Re:Show HN: I'm building an open-source, high-frequency trad... by astrodust on Monday April 15, 2013 @01:14PM (#5552657)
The reason spreads have come down is mostly due to better information on the part of the traders. Electronic markets made this possible.
Re:Show HN: I'm building an open-source, high-frequency trad... by zamalek on Monday April 15, 2013 @08:41AM (#5551110)
"HFT is an interesting multi-disciplinary problem"

This makes this a very worthwhile project: whether or not anything comes of it (I considered writing technical analysis trading software for the exact same reason).

Some documentation about the patterns and techniques you used would likely benefit the community a tiny bit more. Not everyone understands Haskell (or, forbid, has the time to peruse large codebases).

Best of luck to you!

Re:Show HN: I'm building an open-source, high-frequency trad... by xradionut on Monday April 15, 2013 @09:04AM (#5551224)
There are many other interesting multi-disciplinary problems that are not as contentious as HFT and probably more publically acceptable and worthwhile. But if HFT is the domain you care about, go for it.
Re:Show HN: I'm building an open-source, high-frequency trad... by nezza-_- on Monday April 15, 2013 @08:31AM (#5551068)
No, long-horizon investors are normally overall not interested in HFT because they won't 'suffer' from it.
Re:Show HN: I'm building an open-source, high-frequency trad... by dyno12345 on Tuesday April 16, 2013 @12:51AM (#5556316)
If you're already in the finance industry, how do you have enough free time to work on this?
Re:Show HN: I'm building an open-source, high-frequency trad... by white_devil on Monday April 15, 2013 @08:43AM (#5551118)
> an open source approach can be seriously competitive

Do you really think any Joe Schmoe off the street could just grab your open source HFT and start making money with it? If not, how does your project benefit anyone?

Are you working in the industry, but not in HFT? Maybe this is project is just practise for getting a job in HFT? I imagine that's where programmers get the fattest paychecks in the world. Only a fraction of what the sociopaths running the show get, of course, but good money nevertheless.

> I just don't agree that HFT is a scourge.

Good thing you're not at all biased.

Re:Show HN: I'm building an open-source, high-frequency trad... by zygomega on Monday April 15, 2013 @09:06AM (#5551239)
Good points. To be more precise, I think that an open source approach to system research and design can be hyper-competitive versus closed-door secret-squirrel development run by a committee looking for short-term wins. This project benefits me firstly because I get feedback on my initial scratchings and maybe even collaboration on areas outside my comfort zone. And it could well enable me and others to avoid having to work for the sociopaths.

I am biased in thinking that monitoring the market and trading at low latency isn't fundamentally a bad thing, and shouldn't be taxed because others are too lazy to do the same thing. I certainly think that hft has been used by evil people to front-run unknowing third-parties, often in collaboration with middle men and women who turn a blind eye to the morality of their business models.

Re:Show HN: I'm building an open-source, high-frequency trad... by acjohnson55 on Monday April 15, 2013 @09:37AM (#5551376)
Others are two lazy? It's my understanding that the average Joe has no access to the hardware and low latency network connections to be able to do this on his own.
Re:Show HN: I'm building an open-source, high-frequency trad... by zygomega on Monday April 15, 2013 @10:05AM (#5551516)
There is a rather large economic rent attached to HFT at the moment. If my project or others like it can eat into that rent a touch, it might not stay that way.
Re:Show HN: I'm building an open-source, high-frequency trad... by fnordfnordfnord on Monday April 15, 2013 @01:42PM (#5552870)
Exactly, if you take the position that gains from HFT are basically "stolen" profits from otherwise more productive market participants, then any that can be "stolen back" is a win.
Re:Show HN: I'm building an open-source, high-frequency trad... by fbru02 on Monday April 15, 2013 @08:02PM (#5555327)
There have always been "market-makers" why do you care if they are algorithmic or human ?
Re:Show HN: I'm building an open-source, high-frequency trad... by ycombobreaker on Tuesday April 16, 2013 @12:15AM (#5556213)
> It's my understanding that the average Joe has no access to the hardware and low latency network connections to be able to do this on his own.

Yes, but for US equities this is purely economical: _anyone_ can buy colocated rackspace, direct market data feeds, and direct connectivity. The average Joe does not want to do this any more than he wants to build his own tennis-shoe factory.

Re:Show HN: I'm building an open-source, high-frequency trad... by jahewson on Monday April 15, 2013 @09:32AM (#5551357)
Sorry, you're criticising the mindset of "short-term wins" and you're in HFT? What do you think HFT is? Wake up and smell the coffee.
Re:Show HN: I'm building an open-source, high-frequency trad... by mseebach on Monday April 15, 2013 @10:16AM (#5551572)
> Good thing you're not at all biased.

> As you may well be aware, HFT is a scourge on the world's economy, and it's a game only the biggest and best-connected players benefit from.

Good thing you're not stating opinion as fact.

Re:Show HN: I'm building an open-source, high-frequency trad... by scott_s on Monday April 15, 2013 @09:35AM (#5551370)
I don't get your sarcasm at the end of your post. He's stating up front in plain terms what his position is; he's not trying to hide his bias.
Re:Show HN: I'm building an open-source, high-frequency trad... by cmdkeen on Monday April 15, 2013 @08:16AM (#5551007)
HFT isn't necessarily a scourge - if anything it discourages people from short term investing, which is a Good Thing.

Long term investing isn't affected by HFT, except in providing more liquidity at the point you want to exit any position. If you're planning on holding for 5+ years you're competing against relatively few people and can do good research. Any shorter timeframe and you're competing against millions, against computers and against unknowables that will distort the price in the short term.

Re:Show HN: I'm building an open-source, high-frequency trad... by zygomega on Monday April 15, 2013 @08:38AM (#5551098)
Long-term investors trade every day. You wake up and the market has gone down a few percent, you're going to rebalance at least. You decide to buy 30 year treasuries you're not going to hold them for thirty years - you're going to sell 29.5 year bonds and buy 30 year bonds in a few months time.

And managers turn over long-term investor portfolios quickly. The average holding period for an SP500 company shareholder is 100 days.

And every time a trade happens you run the risk of getting clipped by the faster guys who see you coming.

Re:Show HN: I'm building an open-source, high-frequency trad... by cmdkeen on Monday April 15, 2013 @08:53AM (#5551171)
But long term investors don't care about the "clip" that HFTs take, they care about the 80% rise in price over the next 5 years. Rebalancing portfolios again isn't something where getting a few basis points off the HFT best execution price is an issue. I work for a proper long term investor, and we trade over a period of days for the big trades. HFTs aren't even an issue apart from their occasional spectacular blowups that cause "could this happen to us?" memos from upstairs. Long term investors don't need to evolve from HFTs, they're more than happy making money not involving computers or trying to compete with them.
Re:Show HN: I'm building an open-source, high-frequency trad... by ycombobreaker on Tuesday April 16, 2013 @12:27AM (#5556242)
> And every time a trade happens you run the risk of getting clipped by the faster guys who see you coming.

I think it is natural, but fallacious, to apply line-of-sight properties to trading. If you plan to trade, then there are two ways that another participant can "see you coming": 1. If you don't have direct market access, your order gets routed through a broker. The broker sees your order before it hits the market, and if he jumps in line ahead of you then that is front-running, and a Bad Thing. Your broker can get in a lot of trouble for this sort of thing. 2. If you are trying to move a large position by sending multiple orders to the market (one after another), then all market participants have the chance to react to the first order. It's really a game to try to move a lot of inventory at once, without tipping your hand to anyone else in the room. Thems the breaks. Nobody else gets to see anyone's order before the matching engine has already processed it, so there's no way to jump "ahead" of it.

OTOH, maybe your long-term investor is trying to time the market: wait for a signal intraday, and pick that moment to send an order. In that case, if it is a good intraday signal then it is likely that someone else will compete. It is unlikely for a long-term trader to have spent as much on infrastructure as a HF trader, so the juicy signals will result in missed executions that _look_ like front-running.

Re:Show HN: I'm building an open-source, high-frequency trad... by zygomega on Tuesday April 16, 2013 @05:02PM (#5560876)
Yes, very loose terminology. I was thinking of a combination of two events:

1. VWAP based trading of large positions, which creates assymetric momentum effects in volume and price (and which is then somewhat forecastable)

2. Not then recognizing that you are forecastable (as a result of playing the VWAP game). The extra information that someone who looks at intraday price relationships has over someone who doesn't. If you wade into the middle of a market that is short-run forecastable (eg it's trending downwards to a new level and the market maker/HFT guys are battling their battles), and you don't check as to whether it's short-run forecastable, then you're probably the patzy at the poker table.

Re:Show HN: I'm building an open-source, high-frequency trad... by white_devil on Monday April 15, 2013 @08:31AM (#5551065)
It's weird how this "providing liquidity" distraction never seems to go out of fashion. HFT provides liquidity if you happen to be another HFT algorithm. Otherwise, not so much.

Why would short term investing be bad, and how long do you think HFT algorithms hold their "investments"?

I hope you're not just trying to blow smoke up my ass here.

Re:Show HN: I'm building an open-source, high-frequency trad... by cmdkeen on Monday April 15, 2013 @08:39AM (#5551102)
It isn't just about liquidity, though it does. But it also reduces the bid/ask spread, i.e. price discovery. The HFT trader may make a tiny profit when it processes a trade from me, but I've benefited because I can buy/sell at a price close to the quoted exchange price, which didn't use to be the case. In terms of liquidity those trades are available to all, they love buying from non-HFT trades because they can make a small profit from their less accurate pricing. You benefit from the lower spreads and that liquidity. Before HFT you'd have paid a greater cost due to the spread on the price.

Short term investing is "bad" because you're not investing in the future success of the company you're trading on the basis of what you think the market is going to do. To the extent to which that is ever knowable it is unlikely you have the skill, experience and data to be able to beat the large number of professionals doing it.

Re:Show HN: I'm building an open-source, high-frequency trad... by zygomega on Monday April 15, 2013 @08:58AM (#5551194)
Investing in company stock is much more about investing with a global pool of liquid capital seeking risky asset exposures these days. When global capital changes it's mind (in whatever direction) those watching the market closely will be the first to know and the first to profit from the market signal. You can go broke investing in companies that look like they have a bright future.
Re:Show HN: I'm building an open-source, high-frequency trad... by white_devil on Monday April 15, 2013 @08:55AM (#5551180)
It sounds like you work for the financial industry too.

> The HFT trader

You say it like it's a person making trades.

> I've benefited because I can buy/sell at a price close to the quoted exchange price, which didn't use to be the case.

What's a "quoted exchange price" if it's not the price you actually pay?

> Short term investing is "bad" because you're not investing in the future success of the company you're trading on the basis of what you think the market is going to do.

Isn't this what HFT is all about?

Trading on insider information is bad, of course. But if you're on a level playing field, I don't see anything wrong with profiting from something you correctly predicted short-term.

Re:Show HN: I'm building an open-source, high-frequency trad... by TheCoelacanth on Monday April 15, 2013 @09:37AM (#5551375)
> What's a "quoted exchange price" if it's not the price you actually pay?

The actual price you have to pay is based on how much you want to buy and how much other people are willing to sell at what price.

People who want to buy give a "bid" price and people who want to sell give an "ask" price. Whenever there is an ask price that is lower than a bid price, a sale takes place. This results in the lowest ask price always being higher than the highest bid price. The quoted exchange price will be somewhere in between the two prices.

If you want to sell right away, the most you can get is the highest bid price on the books. If you want to buy right away, the cheapest you can get it is the lowest bid price on the books. This means that you have to pay more than the quoted exchange price to buy and receive less than the quoted exchange price to sell.

Re:Show HN: I'm building an open-source, high-frequency trad... by cmdkeen on Monday April 15, 2013 @09:08AM (#5551247)
I do - but a 'proper' long term investor that cares about the price in 20 years time.

The quoted exchange price is an average of the buy/sell offers available, there is no guarantee if you try and trade that you will pay that amount. Especially if you are shifting large blocks of stock.

There is nothing wrong with "short term predicting" but it isn't investing. It is often a very easy way to lose money, "bad" doesn't just mean morally so. It comes down to time horizons, how much you want to make, and how many people are also trying to predict that event. You can correctly predict that event and still not make money as it is already priced in.

Re:Show HN: I'm building an open-source, high-frequency trad... by tome on Monday April 15, 2013 @10:59AM (#5551801)
> What's a "quoted exchange price" if it's not the price you actually pay?

The mean of the bid and the ask, generally. For a daily close price it can be a fifteen-or-so minute average thereof.

Re:Show HN: I'm building an open-source, high-frequency trad... by fnordfnordfnord on Monday April 15, 2013 @01:38PM (#5552832)
>HFT is a scourge on the world's economy

I'd point out that a lot of the things that are going on in HFT, are rehashes of old trading scams, and either are or would be illegal if there was any adult supervision. I was tempted to say something about the SEC being left behind the technology, and doesn't understand it. But for that to be true, and have this crap go on for so long, they either need to be complete fools, or they benefit from the status quo.

>and it's a game only the biggest and best-connected players benefit from.

That's definitely true in the case of latency arbitrage. But here again. It's the exchanges themselves that have decided that some players get an advantage over others. As an example, latency arbitrage, and a whole host of other problems could be sorted out by putting traders on exchange-hosted virtual machines. http://www.dailyfinance.com/2010/06/05/rigged-market-latency...

If not virtual machines, there are other ways to fix these problems, but first you need to get the players to acknowledge that there is a problem.

http://www.zerohedge.com/article/do-it-yourself-latency-arbi...

Nanex has done a lot of good analysis of HFT, some of which is published here:

nanex.net/FlashCrash/OngoingResearch.html

Re:Show HN: I'm building an open-source, high-frequency trad... by ycombobreaker on Tuesday April 16, 2013 @12:46AM (#5556305)
Nanex is wonderful to read, but IMO they frequently misinterpret the data. Off the top of my head:

- They only use consolidated feeds for US equities, never direct market-data feeds. The consolidated feed necessarily contains less information that direct feeds (to satisfy more stringent bandwidth requirements), which masks some "interesting" effects of how the exchanges publish their data.

- They disregard that the CME feed publishes a fixed depth-of-book, and whenever they look at total liquidity in the book it can appear to flicker when deep levels fall "out" of the back of the book, even if liquidity is actually improving with the presence of a new inside level.

- They make a big deal about wholesaler matching only occurring when the consolidated book is not locked. Their rationale is that subpenny prices are always wholesalers, and (erronously) therefore a lack of subpenny-priced trades must mean a lack of wholesale matching.

These mistakes sound believable, but they do not hold up to any of scrutiny. Use their site to find interesting events, but be very careful about taking their conclusions at face value.

Edit: bullet-list formatting

Re:Show HN: I'm building an open-source, high-frequency trad... by zygomega on Tuesday April 16, 2013 @04:51PM (#5560812)
That's why I thought the DNQ iqfeed was a suitable choice - no consolidation at all. Would love to know what the "interesting" effects are (beyond the fixed book CME effect) - will be looking a bit harder at the stream to find them!
Re:Show HN: I'm building an open-source, high-frequency trad... by topbanana on Monday April 15, 2013 @09:06AM (#5551237)
> As you may well be aware, HFT is a scourge on the world's economy, and it's a game only the biggest and best-connected players benefit from. This is the line peddled by the popular press but it isn't true. HFT works to reduce spreads, which benefits both buyers and sellers.
Re:Show HN: I'm building an open-source, high-frequency trad... by fnordfnordfnord on Monday April 15, 2013 @02:36PM (#5553283)
Latency arbing is a scourge, and makes limit orders practically useless. Quote stuffing, would be illegal if trading were still done on little slips of paper. Imagine dumping 10,000 slips of paper on the trading desk, and then shouting "just kidding!" In the words of Lawrence from Office Space, "You'd get your ass kicked" Or even worse, 10,000 empty bids. That's why people hate HFT.
Re:Show HN: I'm building an open-source, high-frequency trad... by kasey_junk on Monday April 15, 2013 @03:39PM (#5553745)
I'm not trying to be argumentative, but I am curious why you find latency arb so problematic, and what you think it's impact is on limit orders?

I for one think latency arb is one of the bigger net wins for hft. As a market participant, each venue I have to maintain a presence at is a cost to me. I'm willing to pay the latency arb shops their cut to provide me price consistency because for my models it is much cheaper to do so than to continually reevaluate and redeploy to every possible venue. It frees me to shop for venues that provide the best features and fees.

As for quote stuffing, you are absolutely right it is awful. That's why almost every venue out there has taken or is taking steps to curtail it. They did so because their customers agree with you.

Re:Show HN: I'm building an open-source, high-frequency trad... by ycombobreaker on Tuesday April 16, 2013 @12:53AM (#5556326)
> I'm not trying to be argumentative, but I am curious why you find latency arb so problematic

That is a loaded term. "Latency arb" as you described it is HFT keeping all protected exchanges synchronized, and it is a good thing. It means that everyone else can ignore the 13 exchanges, and send their orders to the market with the most competitive pricing for connectivity. "Latency" arb" as described in most literature critical of HFT is the specific practice of submitting and canceling non-bona-fide quotes to an exchange with the intent of slowing down the matching engine. If you can slow down the matching engine that most other participants are using, you can effectively delay the public response your actions on the other 12 exchanges. A trading strategy that operates on the basis of a DoS attack on one exchange is definitely problematic.

Re:Show HN: I'm building an open-source, high-frequency trad... by kasey_junk on Tuesday April 16, 2013 @07:18AM (#5557421)
What you are describing is quote stuffing. Nearly every participant in HFT/algo trading agrees that it is a problematic practice. Nearly every venue has either enacted or is enacting policies and procedures to either prevent it or severely penalize it, because that is what all of their customers want.
Re:Show HN: I'm building an open-source, high-frequency trad... by ycombobreaker on Tuesday April 16, 2013 @09:50AM (#5558011)
> What you are describing is quote stuffing.

Yes. Sometimes it is referred to as latency arbitrage. Hence why people _nominally_ talking about that subject may, in fact, be talking about different things.

> Nearly every venue has either enacted or is enacting policies and procedures to either prevent it or severely penalize it, because that is what all of their customers want.

Regarding penalties to prevent or penalize it, I disagree. NASDAQ's policy uses a ratio weighted by distance from the top-of-book. There is no penalty for excessive order submit-cancel loops at the top-of-book. Their matching engine also operates in a fashion which specifically encourages cancel/resubmit loops at the top-of-book, in that they accept and subsequently display limit orders at a different price than submitted. If they were serious about preventing quote-stuffing, they could fix this simply by rejecting those orders. Presumably they either don't care (because their system doesn't get bogged down), or there is pressure from some big customers to keep the status quo here.

Re:Show HN: I'm building an open-source, high-frequency trad... by kasey_junk on Tuesday April 16, 2013 @11:38AM (#5558749)
A) Top of book quote stuffing is a pretty dangerous game to get into. Can't confirm that no one is doing it, but I'd be surprised if it were a big problem. B) NASDAQ is one of the better technology platforms and does not experience as much lag as the others, so it could be as you say, they just don't care. C) the charges away from inside market are only one of several ways NASDAQ discourages excessive quoting. They also have tiered rebate pricing based on quote/fill ratios and PSX has a order rest time requirement.

So I guess if you were a shop that didn't mind playing with fire quoting top of book, and you never actually wanted to market make on NASDAQ, you could still quote stuff them.

I stand by my statement that venues continue to enact penalties to discourage quote stuffing and is such is not nearly the problem people make it out to be.

As a market participant, if the NASDAQ is not providing you with an execution platform to your liking (whether due to laggy matching or anything else) you are free to choose another venue and thanks to latency arb shops you are probably not going to pay much of a price premium to do it.

The biggest problem in all discussions of HFT/algo trading especially when related to internet forums and expose reporting is people using incorrect terms. We don't let people get away with it in other technical settings because it leads to unnecessary strife. I think the same thing applies to electronic trading.

Re:Show HN: I'm building an open-source, high-frequency trad... by fnordfnordfnord on Tuesday April 16, 2013 @10:39AM (#5558337)
Quote stuffing is differentiated from plain old vanilla latency arbitrage by fact that the purpose of quote stuffing is to cause artificial latency to then take advantage of. IMO one can still make defensible arguments in favor of latency arbitrage when they have invested in faster technology than other market participants. There is certainly no defensible argument for causing the latency and then taking advantage of the problem you caused to extract a profit.
Re:Show HN: I'm building an open-source, high-frequency trad... by dchichkov on Monday April 15, 2013 @05:17PM (#5554427)
'Quote stuffing' is there, because of absolutely ridiculous SEC rules. If you'd put in fractional prices and remove NBBO rules, there wouldn't be any 'quote stuffing'. And latency arb. What's wrong with latency arb? How else can you move information from one exchange to another fairly?

Think of HFT's as of part of the financial network infrastructure. Where exchanges play role of nodes and HFT companies role of links/queues/buffers.

Re:Show HN: I'm building an open-source, high-frequency trad... by fnordfnordfnord on Tuesday April 16, 2013 @08:52AM (#5557750)
No worries, I'm specifically talking about the folks that are constantly testing for buys with tiny (and fake) orders and incrementing the ask until they find the limit, then buying all they can, holding for only milliseconds and reselling at the higher price. Queue jumping tricks like that which really add no value.