News for hackers. Stuff that matters.
Re:AI That Picks Stocks Better Than the Pros by rauljara on Thursday June 10, 2010 @02:33PM (#1421123)
Thank you. That was the explanation I was looking for.

My next question then, is whether this flexibility to sell instantaneously is worth the risk of near instantaneous drops of 700+ points in the dow, as happened a couple of weeks ago. My gut tells me that letting computers have a free run of it is not good. But I'm definitely open to being convinced otherwise. And I'm definitely open to a better solution than you have to hold on to stocks for a month or more.

Re:AI That Picks Stocks Better Than the Pros by leelin on Thursday June 10, 2010 @04:21PM (#1421496)
The near instantaneous drop of 700 points is exactly why you need market makers. In normal times, dozens of computer algorithms are posting bids to buy and offers to sell stock, at slightly more profitable prices than what their algorithm determines is the fair value.

On the day in question, the algorithms' "oh crap" mode kicked in, the bots no longer felt they had a grasp on fair value, and decided to withdraw all their bids and offers. Now whoever was left trying to sell (like all the people with STOP orders) had to do so in an environment with very few counterparties.

The stock exchanges give these market maker bots a few special privileges in return for them providing liquidity regularly. It varies by exchange, but one example is a market maker must quote a spread of no greater than 10 cents, and provide both a bid and offer at least 99% of the time. In return, the market maker gets a special spot in line when orders come through (think of it as first dibbs on certain proposed orders).

So it sounds like we should either blame the guys who put in market stop orders, or we should blame the exchanges for not requiring more strict rules for their market makers (like 100% uptime). You can't blame the algorithms for wisely using the 1% of the time they are allowed to hide and wait for more opportune times.

Re:AI That Picks Stocks Better Than the Pros by yummyfajitas on Thursday June 10, 2010 @11:51PM (#1422708)
On the day in question, the algorithms' "oh crap" mode kicked in, the bots no longer felt they had a grasp on fair value, and decided to withdraw all their bids and offers.

The algorithms were not turned off because they didn't work. They algorithms were turned off because everyone knew the exchanges were going to break trades.

If your smart algorithm finds the bottom at 10, buys, sells at 20, and the market goes up to 30, you just made $10, right ($20-$10)? If the exchanges break your trade at 10, you are now short the stock at $20, and the current price is $30! Unfortunately, it's impossible to predict whether the trade will be broken; it's a human making that decision.

Incidentally, most high frequency traders get no special privileges. There are a few that do (the ones you describe), but they are special cases rather than the general circumstance.

Re:AI That Picks Stocks Better Than the Pros by leelin on Friday June 11, 2010 @01:35AM (#1422921)
I agree with you, and in cases where the HFT bot gets no special privileges, then they are under absolutely no obligation to be there when hell is breaking loose. Why is everyone so mad at the bots for playing it safe and deciding not to trade?
Re:AI That Picks Stocks Better Than the Pros by djb_hackernews on Friday June 11, 2010 @09:13AM (#1423720)
I've seen that explanation that people ended up short and deep in the hole after the trade is broken but it just doesn't make sense. Got any links where I can read more?
Re:AI That Picks Stocks Better Than the Pros by yummyfajitas on Friday June 11, 2010 @01:26PM (#1424005)
Re:AI That Picks Stocks Better Than the Pros by djb_hackernews on Friday June 11, 2010 @02:35PM (#1424244)
So basically you have nothing to back your statements up. It appears that when the exchanges break the trade, its completely unwound, as if it never happened. Shares go back to the seller, money goes back to the buyer. No one ends up short.
Re:AI That Picks Stocks Better Than the Pros by yummyfajitas on Friday June 11, 2010 @03:12PM (#1424379)
Nothing you said contradicts what I said. The scenario is that you make two trades - a buy at $10 and a sell at $20. The buy is broken but the sell is not.

Now it's as if you sold but the buy never happened (i.e., you sold stocks you didn't own). That's a short.

Re:AI That Picks Stocks Better Than the Pros by caffeine on Thursday June 10, 2010 @02:52PM (#1421199)
They tested their software on 5 weeks of data from mid-2005, chosen because those weeks have no "atypical" movements. Which means that this algorithm would've been just as cooked as any human being would, no more and no less.

The problem is that there does not (currently) exist any entity or human being smart enough to price stocks correctly. The effect appears so dramatic simply because markets move quickly, and the low transaction costs do little to damp the oscillations taking place.

The "pain" caused by such sudden oscillations is certainly real, and can be thought of as shear stress as the financial system pulls on a slow-moving economy. But rather than arguing for damping the financial system, as you do, I would prefer to lower transaction costs in the rest of the economy. At some point, this becomes difficult: the bottleneck is human beings' capability to respond at high speed (that the financial sector lets computers do most of the trading nowadays is clear admission of this weakness).

The only solution, then, is to let computers make all the other decisions in the economy as well. Hence, I propose a tax on all financial firms, to be deposited directly to the accounts of the Singularity Institute.

(You may decide for yourself how much of the above you consider to be serious..)

Re:AI That Picks Stocks Better Than the Pros by patrickk on Thursday June 10, 2010 @03:08PM (#1421256)
The high frequency trading (HFT) systems that seemed to cause the drop you mention also make it more expensive for ordinary investors to actually buy stock in the first place, thereby minimizing your potential return (as Buffett says, "you profit when you buy" or something to that effect).

There was an article on this a few weeks back where a guy in the know was explaining how such systems are a bad thing. Say you bid a price of $2 per share for a share that is currently at $1.92. Since the advent of HFTs, there's a much higher probability that the price you eventually pay will be a lot closer to $2 than to the current $1.92, (perhaps $1.98 or $1.99) whereas before HFTs you could get it closer to the current price. This matters particularly when you buy a lot of shares, and when you hold them for a long time, as the couple of cents difference compounds a lot over time.

I'm not an expert on the subject, that was my takeaway from it.

Re:AI That Picks Stocks Better Than the Pros by SkyMarshal on Thursday June 10, 2010 @04:23PM (#1421503)
Basically what the HFT algorithms do is probe orders for their market limits.

Say you want to buy your 3000 shares, and you don't want to spend more than $20/share, so you enter your bid with a $20 limit. Your limit is supposed to be invisible to the market, only you and your dealer know it. But the HFT algorithms have a way to probe your order and uncover your limit.

The stock is currently trading around $15, so your order hits the exchange, and in the first few milliseconds, a HFT computer makes a micro-offer for 10 shares at $16, then 10 shares at $17, then 10 shares $20, then 10 shares at $21.

When the last offer is rejected b/c it's over your limit, the computer then guesses your limit is $20 and offers to sell you 2960 shares at your limit of $20.

All this happens in just a few milliseconds, and it just cost you roughly (2960x20 + $20 + $17 + $16 + $15) - (3000 x $15) = 59,268 - 45,000 = $14,268. That's 30% more than you would have paid had you been able to execute at $15.

It's basically cheating, and should be made illegal. Just one more example of how our markets are rigged.

Re:AI That Picks Stocks Better Than the Pros by jrockway on Thursday June 10, 2010 @04:28PM (#1421529)
Of course, if your limit was $15, it would have executed at $15. Or not at all, in which case $15 was not the asking price.

Remember, the price is a two-way street. It's what you are willing to pay, and it's what the seller is willing to sell for. If you say, "I would like to buy at $20" and your partner says "$20 is fine with me too", and the trade happens at $20, then the market is working perfectly. You could have gotten a better deal, but it's not what you asked for.

Re:AI That Picks Stocks Better Than the Pros by jfager on Thursday June 10, 2010 @08:39PM (#1422309)
Let's say you're an institutional investor with an obligation to maintain a particular investment profile, and there's some event that forces you to rebalance your portfolio in a fairly dramatic way. You're used to working with stable portfolios with more or less predictable returns, and this might be a bit much for you to take on, so you decide to blow some of your management fee on some really smart consultants who can do the rebalance for you. They, in turn, have to figure out how to actually execute the plan that they come up with - there are going to be some pretty big orders involved, and if the market gets tipped off that this is going to happen, people are going to start hoarding their shares to drive the price up to screw you. So they start chopping the order up into much smaller orders and dribbling them out slowly, or looking for baskets of stocks that act like the stocks you really want, or trying to fill from dark pools, all kinds of crazy tricks (and they have to do the same kind of things on the sell side to finance all this).

Meanwhile, some even smarter guys at an investment bank just down the road know all these tricks, too, and have come up with a way to algorithmically spot them happening more accurately or more quickly than anyone else. So they start jumping in front of your trades, not because they want the stock, but because they know you want it and are probably willing to chase a rising bid for a while. Then all the other algorithms start to notice, and they start doing the same thing.

You, the friendly investment manager, are now financing both sides of what really amounts to one of the coolest deathmatches going, a bunch of math and computer geeks playing on your quarter and keeping a penny for the honor of letting you watch them do it. But you actually don't care, because it wasn't your quarter in the first place, and now that your fund is back in order, you're not going to get sued, and your customers, who had no idea that any of this was going on, and who may not even realize they're your customers, are back to paying 2% fees for (statistically likely) below-market returns.

In the past, these kinds of large trades would usually end up splitting the difference, with 'real' sellers earning a bit more than what they would have otherwise settled for and 'real' buyers paying a bit less than their actual ceiling. But now 'real' buyers end up paying close to their high bid, and 'real' sellers end up earning their low ask, with the bots fighting it out for the very thinly sliced middle territory.

In one sense, I do agree that this is an example of a market working perfectly. Algorithmic traders are exploiting an arbitrage opportunity between mainstream industry practice and cutting edge technical ability, and just like theory would suggest, they're making bank from it.

Re:AI That Picks Stocks Better Than the Pros by jrockway on Friday June 11, 2010 @09:58AM (#1423879)
In one sense, I do agree that this is an example of a market working perfectly. Algorithmic traders are exploiting an arbitrage opportunity between mainstream industry practice and cutting edge technical ability, and just like theory would suggest, they're making bank from it.

I think this is all there is to be said. Your trading has a pattern that makes them money. The algorithmic traders are going to use that information advantage to make money. That's the idea of the market; trying to extract as much value from securities as possible.

Even if you impose arbitrary trading limits, the big banks are still going to find a way to make money off of you.

Re:AI That Picks Stocks Better Than the Pros by jfager on Friday June 11, 2010 @10:24PM (#1425433)
There are lots of ways to make money that would ultimately harm the market that have been made illegal or have been banned by exchanges. Nobody has a problem with people getting rich from providing value, but there's a long history of patching the system when it looks like someone's starting abuse an advantage that doesn't return any value to the market.
Re:AI That Picks Stocks Better Than the Pros by SkyMarshal on Thursday June 10, 2010 @08:12PM (#1422252)
The problem is, you're not telling your partner 'I would like to buy at $20'. You're telling him 'I would like to buy at $15', but as in any negotiation you have a max price you are willing to pay and he has a minimum price he is willing to sell, but neither of you announces that to each other.

However, he basically has the ability to read your mind and see that he can negotiate you up to $20, while you lack the same ability to reciprocate.

It's an unfair advantage enabled by large firms having the money to buy better access to market mechanisms and powerful enough computers to exploit them.

Re:AI That Picks Stocks Better Than the Pros by BrentRitterbeck on Thursday June 10, 2010 @08:44PM (#1422316)
This is similar to me saying that it is unfair for a 6 foot 7 inch person to play offensive tackle in the NFL when I am only 6 ft 1 inch and would like to play the NFL as well. The world is not fair. Some people will have the resources to do certain things while other people will not.

EDIT: Before you go on to say that football and investing are two very different things, let me state that I am fully aware of that. I've had that conversation before when this topic comes up. Common people are able to participate in the investment process while only a very few get to play in the NFL. The people that are upset are the ones that try to game the market themselves but are under-capitalized. From prior conversations, long-term investors simply don't care about these things. The people all up in arms are the one's that have lost money because they are trying to do something they are not in a position to do. I'm only talking about trading stocks and futures here; the more advanced investment products, like CDOs and CDSs, and the issues in the news regarding them, are a totally different conversation.

Re:AI That Picks Stocks Better Than the Pros by SkyMarshal on Thursday June 10, 2010 @09:37PM (#1422430)
It's not similar to that at all, or even related in any way. Without transparency and fair mechanisms, markets break down.

Participants getting exploited withdraw and find other ways of getting what they need, most recently through dark pools.

Crucial liquidity diminishes, and eventually all that's left are the crooks. The suckers are broke and honest non-suckers are gone.

If you want functioning capital markets, you have to regulate and crack down on stuff like this.

Re:AI That Picks Stocks Better Than the Pros by patrickk on Friday June 11, 2010 @10:03AM (#1423893)
Exactly. I believe that HFTs are not just unfair but do not allow the market to fulfill it's basic premise as effectively.

The market is supposed to be fair and a level playing field for all those who can afford to buy stock, but the HFTs tilt the odds in favor of the select few.

When the odds are tilted in favour of a select few, it becomes a problem for all. The bond markets are far less regulated, thereby allowing Wall Street to profit massively from confusion and a lack of regulation - Wall Street makes most of it's profits on the bond market. The result is that insane toxic assets get sold, all because the industry can supposedly self-regulate. Result: global catastrophe. (See "The Big Short" by Michael Lewis for a great read on how Wall Street caused the financial crash of 2008).

Re:AI That Picks Stocks Better Than the Pros by BrentRitterbeck on Thursday June 10, 2010 @09:53PM (#1422466)
You say that it is essential to crack down on high-frequency trading. On what grounds? The markets function because these firms provide liquidity. If you want to see what happens when these firms stop providing liquidity, you can look at the flash crash. To crack down on the industry in the ways being proposed would have the exact opposite effect that the regulators are looking for. Spreads would widen. Volatility would increase. Volume would severely drop. Congress has no idea that they are legislating the next financial crisis into existence.
Re:AI That Picks Stocks Better Than the Pros by SkyMarshal on Thursday June 10, 2010 @11:47PM (#1422702)
Oh right, I totally forgot there was no liquidity in the markets before HFT came along. How silly of me.

The flash crash was exacerbated by HFT, not caused by the absence of it.

Congress doesn't have to outlaw HFT, but it does have to outlaw HFT order limit probing. Though it would probably not hurt to completely unplug HFT anyway.

There is zero net benefit to the markets from it, and there will be plenty of trading and liquidity without it (just as there was before it). Arguably even more, as funds that now resort to dark pools return to the open markets.

Volatility spikes would still exist (always have always will), but HFT exacerbates them, lack of HFT mitigates them.

The problem with HFT is that it is not a productive enterprise. It does not take raw materials like copper and sand, and apply labor, capital, and innvotation to turn them into products worth more than the sum of their inputs, creating real wealth. It doesn't mine or farm those raw materials. It doesn't even provide a specialized service that saves people time/money.

All it does is skim off the top of the global capital flow, increasing the cost of that capital to everyone else, particularly to the productive enterprises mentioned above. It is parasitic, and gives nothing back, despite the 'story' Wall Street has created to justify it.

Re:AI That Picks Stocks Better Than the Pros by BrentRitterbeck on Friday June 11, 2010 @12:06AM (#1422743)
I never said there was no liquidity in the markets before HFT came along. I was making the argument that there is much more liquidity because of high-frequency trading. As a result of this much more liquidity, the markets are more efficient. If your only concept of a productive enterprise, however, is one that produces material goods, then there is no point in having this conversation. I do have one response to your sentence:

It doesn't even provide a specialized service that saves people time/money.

You are so far from reality here. Do you understand what the futures markets are for? Do you understand that there are a whole class of speculators, among them are HFT firms, that stand ready to do business with a farmer growing corn, a gold miner mining metal, a cattle producer with cattle to go to the market, etc? By providing liquidity, the farmer can focus on farming, the miner can focus on mining, and the cattle rancher can focus on ranching (whatever they call it) instead of worrying about what they may sell their raw goods at in the market four, five, or six months from now. So, no, I am not out in the fields digging up copper and sand, but I, and others in the industry, make it possible for those that do those things to focus on doing those things. If I figure out a way to make some extra money by using mathematics and computer science, then so be it.

Re:AI That Picks Stocks Better Than the Pros by clammer on Friday June 11, 2010 @01:15AM (#1422884)
Is it me, or are there a lot of communist on HN? Such fear and loathing of free markets. Everyone should be equal. We should all get a 5% return.

We should cease all activities deemed "non productive" by the state and get back to work in the mines.

Re:AI That Picks Stocks Better Than the Pros by patrickk on Friday June 11, 2010 @10:06AM (#1423899)
Being critical of HFTs doesnt make you a communist. It makes you a realist - as the commenter said, HFTs add no real wealth.
Re:AI That Picks Stocks Better Than the Pros by yummyfajitas on Friday June 11, 2010 @12:12AM (#1422757)
This is nonsense. When you place a limit order, it is made publicly available to everyone with a market feed. Everyone with a market feed is informed that someone out there (no name is given) wants to buy 3000 shares at $20 or less.

If anyone out there had a sell order at less than $20, their orders are immediately matched to yours. Lets say someone wanted to sell 1000 shares at $15.1, someone else 500 shares at $15.2, and nothing above that. So you just bought 1500 shares at $15.1 and $15.2. Now there are no shares available to buy below $20, but your order is only half filled.

What happens next is that an HFT system will rush in with a sell order of 1500 shares at $20.0. Your order is filled within a second, and at the price you asked for. Without the HFT system, you'd need to wait a few seconds for a human to come along and fill your order at $20.0.

Re:AI That Picks Stocks Better Than the Pros by BrentRitterbeck on Friday June 11, 2010 @12:19AM (#1422772)
Limit orders are not hidden at all. There's a 200 share bid sitting in the GE order book right now at 14.28 on Yahoo! Finance. There are six orders sitting in the BAC order book on Yahoo! Finance. If they are suppose to be hiding limit orders, BATS, the provider of the data, is certainly not doing a very good job.

Now, there are certainly means of hiding one's intention. There are also means of finding out where hidden liquidity exists. I will admit to that. Look up icebergs and strategies to detect icebergs. Just because people attempt to hide their intentions while some try to discover those intentions, that doesn't mean there is anything suspect taking place. Information, or the lack of information, is important.

Re:AI That Picks Stocks Better Than the Pros by dkasper on Thursday June 10, 2010 @07:28PM (#1422147)
If you put in a limit order for $20 you should expect to pay $20. By definition, a limit order protects you from paying more, it doesn't guarantee you will pay less or execute at all.
Re:AI That Picks Stocks Better Than the Pros by SkyMarshal on Thursday June 10, 2010 @08:18PM (#1422266)
No, you should expect to pay up to $20. Sometimes $20, sometimes less.

Under normal market operation, sometimes the market will trade upwards during the time it takes to fill your order and you'll pay in a range from whatever the market started at up to your limit.

Sometimes the market will trade down during the time it takes to fill your order and will never hit your limit, especially if you make the order while there are more sellers than buyers (say BP after the accident).

So over the course of many trades you may be able to expect average execution prices around where the market was initially trading when you placed the order, since sometimes you'll pay nearly exactly that, sometimes you'll pay more, and sometimes less.

But as these HFT limit probing scams become more widespread, execution averages will skew toward being more expensive for the party on the other side of the trade from the HFT, which is typically big mutual and pension funds that deal in millions of shares, or Joe Six-Pack buying through some less-sophisticated Etrade or whatever.

Re:AI That Picks Stocks Better Than the Pros by graphene on Thursday June 10, 2010 @04:19PM (#1421483)
That's quite different from the way I understand it.

You have to specify more exactly what you mean when you say the current price is $1.92. Is it the current best (highest) bid, best (lowest) offer, or last traded price (this is generally the displayed price, but there is no guarantee that you can buy or sell at that price, since that depends on current outstanding offers and bids.

Suppose the last traded price is $1.92, and the current best bid (highest price anyone is willing to pay) is $1.90 and the best offer (lowest price anyone is willing to sell) is $1.95. You can either decide to buy from the best offer or bid higher than the current best bid and hope to find a seller.

Where HFT comes in is in that the increased amount of money and number of participants in the market tends to cause the difference between the best bid and best offer prices ("the spread") to go down, making it cheaper for everyone to enter and exit the market as they wish. This is the market making effect hugh3 referred to above.

Please note I am not an expert and may turn out to be wrong on some of this.

Re:AI That Picks Stocks Better Than the Pros by BrentRitterbeck on Thursday June 10, 2010 @08:27PM (#1422286)
If you still pay $2 for a share, is that not the going market price? You don't have to submit a market order. If you wanted the shares at $1.92, you could have submitted a limit order. This is all supply and demand; it's just price discovery happens at a much higher frequency. It is no different than being charged extra for staying at a hotel during the holiday season. The price changes just happen much more frequently and much more quickly.
Re:AI That Picks Stocks Better Than the Pros by th0ma5 on Thursday June 10, 2010 @04:17PM (#1421472)
Rolling Stone had a great article last year, and their opinion was that once you boiled everything down, it sure did seem a lot like front-running.
Re:AI That Picks Stocks Better Than the Pros by gxti on Thursday June 10, 2010 @09:16PM (#1422391)
Matt Taibbi has already proven that he can't tell a short sale from a hole in the ground. Perhaps Rolling Stone isn't the best place to be learning about markets.

cf.: http://www.businessinsider.com/john-carney-matt-taibbi-falls...

Re:AI That Picks Stocks Better Than the Pros by hugh3 on Thursday June 10, 2010 @02:40PM (#1421153)
Well, the near-instantaneous drop we saw a few weeks ago is something that has happened exactly once in history. It will probably never happen again, at least not in the same form, since people have probably gone and changed their software to avoid it. And its consequences, in the end, weren't all that bad -- mostly a bunch of institutional investors made some money at the expense of some other institutional investors, but I haven't heard of anybody getting wiped out.

So I'm inclined to pick the free-market devil I know over the government-controlled devil I don't.

Re:AI That Picks Stocks Better Than the Pros by joe_the_user on Thursday June 10, 2010 @08:23PM (#1422275)
America's greatest period of growth, the 1950-1973 expansion, took place within a highly regulated financial system.

The "devil" of returning to that level of regulation hardly seems unknown.

Re:AI That Picks Stocks Better Than the Pros by djb_hackernews on Friday June 11, 2010 @09:15AM (#1423729)
correlation causation blah.

playing devils advocate, imagine how much more we would have grown had the financial system been unregulated.

Re:AI That Picks Stocks Better Than the Pros by joe_the_user on Friday June 11, 2010 @06:50PM (#1425020)
My point was that a regulated financial system is hardly an unknown, whether "devil" or not. Indeed, high frequency trading seems like far more of an unknown and giving good evidence so far of being a "devil".
Re:AI That Picks Stocks Better Than the Pros by BrentRitterbeck on Thursday June 10, 2010 @08:21PM (#1422273)
The thing with the flash crash is this: the market dropped 1K points immediately because the computers and their operators refused to provide liquidity when the initial suspicious trades hit the wire. While the event may have been caused by a computer, the extent of the drop was more because computers pulled themselves from the market than because high-frequency trading is somehow financially evil. I think people need to understand that a lot of the high-frequency trading actually helps to stabilize the market. What you saw during the flash crash is what happens when liquidity dries up because the computers are not trading.