Even in regulated markets there are problems. I have a friend who built an HFT algorithm that he was using for trading stocks. He had some really good results with early testing. But at some point he was confused why many of his buy/sell orders weren't being executed despite being open for several minutes (hours?) his algo would make bids far away from the current spread, anticipating movement. He finally concluded that some institutional traders must have access to sub-penny ordering, despite it being against regulation.

I didn't believe him at first, since, the more likely problem was elsewhere, but then a few months later sub-penny trading in dark pools was all over the news. This was like 5 or 6 years ago. He's since moved on to other things having come to similar conclusions, that trying to play such a rigged game was futile.

> But at some point he was confused why many of his buy/sell orders weren't being executed despite being open for several minutes (hours?) his algo would make bids far away from the current spread, anticipating movement

* Did his levels actually get crossed? It could be that he never had the opportunity to get filled at the price he hoped for.

* What was his queue position? Stock exchanges tend to be price time ordered. If others had submitted orders at the same price before he had, they would have priority when getting filled.

* Apropos the prior point, was his broker actually submitting orders when he sent them? Some brokers may avoid showing deep out of the money orders, which could have affected his queue position.

> He finally concluded that some institutional traders must have access to sub-penny ordering, despite it being against regulation.

There's nothing wrong with this. Reg NMS rule 612 permits sub-penny price improvement [0].

[0] https://www.sec.gov/divisions/marketreg/subpenny612faq.htm#q...

It's a fact that Joe Blow will get sub-penny price improvement at random. I have no idea how often or how long it's been a thing, but it happens.

I've been getting fills out to four decimal places for a few years now. (Schwab and Interactive Brokers' smart routing)

you can do sub-penny price improvement but you cannot quote at a sub-penny.

Technically this is incorrect. Stocks trading below $1.00 may be quoted in sub-penny increments.

Really interesting, thanks! Could you share bit more about the 'institutional access' call? Does it mean that selected few have access to real-time book updates and all the rest is @100ms delayed? So much of level playing field :) Is it common for asian crypto exchanges?

> the orderbooks were slightly different between bots

That sounds like a big deal. If this is repeatable, you should document it better. Unregulated doesn't mean a license to do blatantly illegal things. Crypto exchanges certainly get taken to court.

Sure it wasnt your book building algo and a snapshot retrieval race?

There's a much less nefarious explanation. I'm not saying they didn't manipulate the order book feeds. I work in the regulated trading space. I've written market data decoders. One is in production inside of a bigger trading system I wrote at a customer as I type this.

The less nefarious explanation is simple. You decide how likely it is: there is no mechanism making sure that IP packets sent to all data feed subscribers arrive at the same time. Exchanges of consequence distribute market data via UDP multicast, over physical links that are as identical as possible (think identical lengths of fibre).

Now if you're receiving JSON via Websocket and parsing it using an allocating parser and your NIC driver is in kernel space and you use a GC'd language and if the exchange loops through a list of TCP connections to send a message to them one at a time and there is jitter in packet delivery time in upstream hosts (and other internet weather) and and and ... you simply cannot expect identical order books at the submillisecond timescale.

A fortiori, throw half of that crap away and suppose they were using userspace NICs and no-alloc single threaded C++ that chills entirely in L1 cache. Still consuming TCP over the public internet.

Over the public internet its likely going to be a TCP feed. I would expect OP was capturing that feed and doing the analysis offline. Most firms do this, even with the collocated UDP setups.

After 2 weeks of digging through our code I’m 100% confident it wasn’t our code. Was able to hook up to a total of 3 additional accounts via api, all 3 books matched, and were started at different times.

We wrote support and they told us they would investigate. Never heard back.

Binance is in Malta, a somewhat corrupt tax haven; are you sure it's illegal there?

Malta is in the EU.

This doesn’t guarantee much in itself. A journalist was murdered there for investigating corruption not that long ago.

And the prime minister’s chief of staff was just arrested over that. Just because illegal stuff happens doesn’t mean there aren’t laws against it.

Sure. That's not the point. The point is corruption increases risk of doing business regardless of whether somebody gets caught. People dying because of investigating corruption only makes the point stronger.

OK, so which EU directive are we dealing with here? Which regulator does Binance answer to?

Their local financial regulator. But the point is EU directives force harmonization of laws governing financial exchanges.

Ran an ML model years ago had a number of great months then out of no where no trade I or the ML would make, would work. Looked like someone was front running my orders and messing with my trades. Weird delays, trades would take to long to go through all, and all sorts of odd events on Level 2. Ended up shutting it down took a good 2 months before my manual trades started going through at a normal rate again.

This is why the whole $0 trading fee and robinhood concern me. I'm paying for the trades and someone is still messing with me.

I think, though I'm not sure, that RH makes their money by investing in treasury bills with the cash balances of people's portfolios. You're (most likely) not getting secretly screwed.

Contrary to popular belief robinhood makes a good bit of money from selling order data, more than most brokerages. This is due to the fact that most robinhood users are inexperienced, and thus, their trades are less likely to indicate big movements of institutional money. I.e. HFTs pay more for data that is less risky.

Edit: Fun fact, the founders of Robinhood worked at an HFT before starting robinhood, which used to be shown on their LinkedIn until several years ago. Bad optics I suppose.

Robinhood sells order flow https://www.cnbc.com/2019/04/18/a-controversial-part-of-robi...

Pretty much everyone sells part or all of their flow to Citadel or Citi.

A great article on the subject: https://www.kalzumeus.com/2019/6/26/how-brokerages-make-mone...

Why invest in treasury bills at 1.8% yield when you can invest in margin lending at 5%?

What's the value of HFT? If exchanges were required to add a random delay to very trade to work against high frequency traders, would anything of value be lost?

If exchanges added a random delay, that would mean traders have to plan for a longer time horizon, which means they need to offer more conservative prices. That means when you're selling, you get less money, and when you're buying, you get less stuff.

Some people have this notion that if their trade got matched against a HFT that they have somehow lost out. The exact opposite is true. If the party on the other side of your trade was a HFT, then that implies that all other parties were offering worse prices than the HFT was. If the HFT had not been there, you would have got a worse price (whether on the buy side or the sell side). The presence of high-frequency traders (or any traders, for that matter) reduces spreads and increases liquidity.

The only people who get worse deals as a result of high-frequency trading are people who would sell you the same stuff for more money.

Hang on, that is conflating separate ideas.

The substance of your post is describing price discovery - price discovery isn't the issue here. The issue is that the price discovery is being done in milliseconds.

The substance of the complaint is the speed of the auction that discovers the price. The faster price discovery is being done, the fewer traders can participate. If a few milliseconds can change the value of a financial instrument then we have to accept that waiting a second or two will allow a lot more traders to reevaluate and offer a fair price.

The HF in HFT looks like a play to reduce the number of traders who can act on information, which means the buyer/seller is probably getting scalped. HFT traders are making money arbitraging the speed of information dissemination, which indicates that other traders would offer different (/better) prices if the market waited a half-second or so to let everyone gather all the relevant data.

> The substance of the complaint is the speed of the auction that discovers the price. The faster price discovery is being done, the fewer traders can participate.

The faster the price discovery, the shorter the validity time. This means a more accurate price.

> If a few milliseconds can change the value of a financial instrument then we have to accept that waiting a second or two will allow a lot more traders to reevaluate and offer a fair price.

You're making the mistake of assuming the price is static. If I have to quote you a price on an instrument that's valid for the next 5s, I have to be more conservative than if I'm quoting for the next 5ms. Which means you get a worse price.

> The HF in HFT looks like a play to reduce the number of traders who can act on information, which means the buyer/seller is probably getting scalped. HFT traders are making money arbitraging the speed of information dissemination, which indicates that other traders would offer different (/better) prices if the market waited a half-second or so to let everyone gather all the relevant data.

There's literally nothing stopping or limiting the numbers of players operating at this speed. What the race actually results in is all market participants getting smaller spreads and better prices.

"The faster the price discovery, the shorter the validity time. This means a more accurate price."

A continuous market price is an illusion; let's not forget that. Every market is made up of trades, which are discrete, and every price has an unstated amount of uncertainty, which may be large by any standards. You can't know from first principles whether a change in quoted price is even a change in the market, because it could be within the +/- range that's implicit. Apple is quoted to the nearest penny, at least, but the idea that the current price is accurate to within 0.004% is ridiculous.

Edit:

"Fischer Black famously defined an efficient market as “one in which price is within a factor of 2 of value, i.e., the price is more than half of value and less than twice value,”

...I hadn't heard this before Matt Levine mentioned it, but I was like "yeah, obviously, why haven't most people gotten the message?"

But: "BNP Paribas...said [Saudi] Aramco was worth exactly $1.424394 trillion."

I wonder which particular millisecond that held for.

Edit Edit: The market has a whole (mostly) unseen dimension other than uncertainty, which is depth. You can only buy or sell so many shares at the instantaneous market price. Further away, there may be orders, but the price of the whole company is going to be way outside of that. The shorter the timescale, the shallower the "market" so you can't just say we're making progress by doing things faster. It's like when research lasers are said to make unbelievable power, but it's like for a femtosecond or something. Liquidity, in my mind, requires depth, just as with water.

I don't believe liquidity necessarily has anything to do with efficiency, but more liquidity can never hurt. Even if the price of every stock right now is 100x what it should be, I still want lower spreads.

And to address your concerns about depth, I believe that's where market makers (which are related to HFT) come in.

Everything is about costs and benefits. It doesn't make sense to say "I want an improvement in X, no matter how small" without regard to cost, because as X becomes tiny, a very small cost could mean a net loss to society, and multiplied by lots of people, is a large loss.

Look at gas stations. If people are spending $1 driving around to save $0.10, then that's not good and at least public policy shouldn't encourage it.

> If a few milliseconds can change the value of a financial instrument then we have to accept that waiting a second or two will allow a lot more traders to reevaluate and offer a fair price.

I'm not quite getting this point. If I have a sell order at price x and it's filled by a HFT 1 second before someone with a slower algorithm, how does that result in a more "fair" price for me? If the HFT instead posts a buy order at an unfair price x-1, there's nothing stopping the slower traders from taking my sell order at x one second later.

That would be totally boring trading and unremarkable, but that isn't what HFT firms are doing. If that were the strategy then exchanges would presumably just sell a skip-the-queue service when two traders offer the same price and there wouldn't be much point worrying about how far machines are from the exchange.

HFT trading isn't about executing the same trade as someone else but a tiny margin faster. That wouldn't have any special impact on market spreads or liquidity, for example.

There aren't any complaints against the T in HFT; as traders they are helpful. The value questions are about the HF and whether it is a desirable part of the market or an unhelpful arbitrage opportunity created only by implementation details of the exchange.

Many HFT systems provide liquidity. If there is no liquidity, retail investors like you or me cannot buy or sell.

Just imagine you want to exchange a currency because you go traveling and the exchange tells you "Sorry, nothing available right now, gotta come back in a few weeks". That's what would happen if there is no liquidity.

I always had questions about the liquidity on offer though seeing participants flee the market in black swan events. HFT liquidity could be illusionary - its only there when its not quite required similar to how the bank "only offers you a loan when you don't need one". Of course this is exactly the point when liquidity is required; normally there's sufficient liquidity in normal times from market participants. Its easy to offer liquidity in normal times; harder to do so when no one else wants to offer it.

Offering liquidity is not equivalent to offering free money. The offer comes at a cost.

> normally there's sufficient liquidity in normal times from market participants.

Not really. The spreads were terrible before HFT market making.

It isn't obvious that the retail investors are that excited by sub-second liquidity.

Speaking personally, I'd even be happy to wait several seconds to see if someone else is willing to pay a better price.

That's not how things work. You would see much less demand/supply for a particular stock at every price level, which means you'd pay much more to buy less and would have to pay much more to sell less also. There are no upsides to having fewer market makers.

> you would see much less demand/supply

Why? If the HFT firm was willing to offer me $X 2 milliseconds ago they are probably still willing to offer $X now. It isn't like there has been time for anything to change; there are going to be short periods of time where there is literally no new information.

And they are just as likely to be offering me more now than less as conditions change.

if they weren't able to move their price 2 milliseconds later, they wouldn't offer $X in the first place.

We obviously can have liquidity on in-between timescales, because we had it before modern HFT trading, right?

The average person may be overly paranoid about HFT, but it doesn't make sense to say they benefit from it, because they are not going to be in a position where they benefit from an execution in a fraction of a second.

Price improvement of fractions of a penny has gotten silly too. It's easy to think of it as more significant than it is, until you figure it as a percentage (or the spread for that matter).

It's kind of like how ultra-sensitive people are to gas prices...

Before HFT, we had enormous spreads and expensive trades on many instruments in many markets.

Smaller spreads lower costs for everyone: institutional, retail etc.

When exactly are you thinking of? Are you old enough to remember before decimalization? Do you consider that "enormous"?

Yes, adding delay generally lowers the economic value of markets. In addition to improving liquidity, as mentioned by others here, HFT also improves the relative pricing of related instruments. There is an enormous number of financial instruments traded for the purpose of prioritizing resources, transforming assets, and offsetting risk. Equities and cryptocurrencies are a small part of this connected financial universe. There are many trading entities with unique information and/or understanding about different parts of this constantly changing universe of prices. None of them has a complete view, but by communicating with each other through trading, their expertise is pooled. HFT speeds this communication network, which improves the dynamic consensus.

As sibling said, it's liquidity. Worked at an HFT for a while, and they once got in a snit with an exchange over some policy change. They turned off their trading there one morning, and within 20 minutes, the exchange was on the phone begging them to come back.

The liquidity from bots has replaced the human brokers. Sell side stock trader is a job that mostly no longer exists Eg https://www.bloomberg.com/news/articles/2018-04-30/goldman-t...

Not really. The real-world value of Google (or any other company) doesn't change meaningfully on a millisecond basis.

Liquidity, which honestly has very little value to a traditional stock market. HFT and day trading both just exist as rent seeking behaviors on company valuation growth.

When You do this kind of (HFT) trading, how does ones tax returns look like? E.g. I assume you still have to report every trade with its gain/loss to a tax authority?

IANAA, but you file a gigantic "Schedule-D". I have never done it with crypto, but there is standard software to do it for non crypto stuff.

I used GainsKeeper: http://www.wolterskluwerfs.com/tax-reporting/solutions/gains...

I see they do crypto, but I've never used it for that.

For companies, they have entire departments that consolidate trades and construct massive Schedule Ds

Depends on the nature of the trades. If you are hedging and maintaining a delta-neutral position (very common situation for market makers and arbitrageurs) then technically your trades are part of a straddle, which is reported separately from capital gains. (in the U.S., at least)

I learned this because I had a six-figure capital loss on my spot crypto trades, and a slightly larger gain on unregulated futures which do not fall under capital gains. Capital losses can only offset up to $3k in other income, so I was terrified that I’d have to pay taxes on the massive “gain” without being able to count the losses against it. Fortunately, the straddle is the correct way to report this, allowing gains and losses to be matched across instruments that may otherwise fall into different income categories.

> It was in that moment we realized playing in unregulated markets is not fun or something we wanted to continue to do. Intermediary risk was something we didn’t account for.

Indeed. This is the really difficult thing about the crypto space: winnings, if you can keep them. And you can't if the house is just going to front-run all your orders.

Fascinating story, thanks for sharing!

"we realized that there will always been a better resourced or more dedicated team willing to fight you for your alpha" - that's part of what makes financial markets such a fun and interesting challenge but agreed, intermediary risk at the timescales you were operating at in this kind of unregulated market is real and not fun

I'd love to hear what problem you moved to that you believe you can build a long term competitive advantage on if you can talk about it?

My partner and I had built broadbandnow.com and were looking for a new interesting problem to solve to get us fire up again about fun tech problems. This just scratched the itch. The team was running day to day of the other business while we worked on the HFT bot.

Shortly after we realized we either should exit or hire an exec team to run the business. After a few months of executive search we found an offer we liked and took it.

These days I’m interested in using some heuristics based on public data to help consumers make informed decisions about nursing homes and in home health care. Still early on this project but lots of data and not many people looking to do good in that market. Seems like a great place I can add value.

Very interesting. Your nursing home project sounds like you could definitely add loads of value - best of luck for it!

> Best we could tell someone was front running us due to an artificial delay for our account (delay between trades went to ~20ms up from our prior steady speed of 3-7ms) and/or a bunch of the trades in the orderbook were bogus.

I wouldn't be surprised, given that traditional HFT companies are building cryptotrading desks and they have a lot more capital to play with too.

You can't simply front-run someone else by having deep pockets.

The only way that is possible is if you gain privileged information about that person's orders before they actually hit the book. The most practical way to do this is to be the exchange.

Or to pay the exchange a large sum to get privileged access :)

Seems they just simply could sell a faster connection and just call it an "institutional feed".

Regulated financial exchanges basically do this already.

> You can't simply front-run someone else by having deep pockets.

Well you can, as was discovered in the Haim Bodek HFT shit storm.

> the orderbooks were slightly different between bots

As somebody who still runs a profitable bot on Binance I find this hard to believe.

Also all order book related endpoints/streams are public, so queries/subscriptions are not tied to a specific account.

In late 2017/early 2018 when their websocket was new I believe you had to be authenticated for either the initial order book query or the websocket subscription. Either way since we always locked our api keys to our IPs it wouldn't have mattered.

Thanks for sharing. Do you think building the strategy on another exchange such as Coinbase Pro or pursuing a strategy that wasn't as latency-sensitive might've yielded more success?

I ran a triangle trading arbitrage bot on Coinbase that was profitable at the end of 2017. But a few weeks later, it was losing money on hard to diagnose slippage. The number of “opportunities” also collapsed, which indicates that other bots were doing the same or that coinbase was doing this internally.

I pulled the plug. Tried to run it on Binance but the websocket only updated once a second, so there was way too much risk.

I have had a similar experience. The house always wins.