Practically, most people don't actually have that many major problems that need solving, and it is pretty simple to invent something that isn't what someone needs, and convince them that they need it, then to come up with novel ideas.
Industry and free enterprise does not solve problems much anymore. It sells the process of convincing people they have problems.
That said, actual product companies are making a comeback in Silicon Valley, and are using innovative new tech like 3D Printing, and there's some amazing work going on in Virtual/Augmented Reality as well (Occulus Rift, CastAR, Google Glass), - all physical products.
And lets not forget Tesla upending the world's perception of the economics of producing and owning electric vehicles.
While I agree with the sentiment about smart people wasting their potential chasing cash, I think it's only an intermediary phase before the next wave of "insanely great", life-changing products hits critical mass.
Citation? The world's perception of Tesla is arguably: expensive, unaffordable, impractcle plaything for rich people. But yeah I would drive it if you paid for it...
In fact it wouldn't surprise me if some sophisticated neural net algorithm developed for HFT was repurposed and used successfully for other things - medical research, climate modelling, brain research - basically anything involving non-linear dynamic systems.
Only problem of course is that all those HFT algorithms aren't FOSS, but the incentive to develop them in the first place would be greatly diminished since they'd lose whatever edge they had almost immediately after publishing.
The markets are a pretty decent lab for experimentation and learning about complex systems, which we're still not so great with, as long as systemic risk can be prevented and every player forced to take their losses instead of passing them off to governments. I personally don't think HFT is remotely as much a problem as what caused the financial crisis - a few flash crashes here and there plus an "HFT tax" on asset prices, vs the arguably-fraudulent origination and dissemination of trillions of dollars of bad debt resulting in the near total annihilation of the global economy. I think we'll gain a lot more (or at least lose a lot less) as a society by prioritizing the latter problem.
Well, smart people's labor isn't your property to allocate. Nor is it the property of "us." It belongs to those individuals.
So it's not a question of what "we" can or cannot afford.
--
Leaving aside for a moment the question of the value of HFT, this is a skewed view of the world of finance. The equities markets may be the most visible to the public (quoted on the news, associated most directly with companies and products we all know, etc), but it's just one market.
In fact, the size of global bond markets is almost double that of equities [1] and is a huge source of funding for companies (and governments), and has more frequent new issues for the same entity than in equities. Similarly futures markets are hugely active and important as both a source of price discovery for commodities and as a way for businesses small and large to hedge risk.
I mention all this not to pick on you in particular, but because it seems in the wake of the financial crises of the past few years, this "finance is bad" mentality seems to be pervasive. There are absolutely problems, but I think it's important not to lose sight of the fact that much of what financial markets do is hugely important to modern businesses and governments.
[1] http://finance.zacks.com/bond-market-size-vs-stock-market-si...
Obviously regulation doesn't seem to do much good because it just changes the rules which sends the investments games scrambling for new arbitrage opportunities which no regulatory agency has the resources to keep up with until its too late. Given the lack of political fortitude to combat moral hazard by forcing financial institutions to take their own losses on the chin, I think perhaps widespread and rabid investment banker hate could be the best societal medicine we have against the brain-drain to finance.
http://www.sifma.org/research/statistics.aspx
It looks like the US bond market daily trading volume is about 25 times higher than the US stock market daily trading volume. However, that statistic is highly skewed by trading in US treasury bonds, which arguably should be a separate category. The same would presumably apply to the stats on market cap.
Also, it's not the relative size of the equities markets that's the problem; it's the amount of effort and talent that goes into what is essentially a zero-sum activity. Bonds and futures (and, to be fair, new stock issues) are positive sum. HFT is not.
(More precisely, HFT is only positive sum to the extent that it improves price discovery and therefore market efficiency; but I have a really hard time seeing how shaving a few more microseconds off trade times changes that significantly. It does, however, significantly change who is able to benefit from asymmetric information by inducing others to take the wrong end of zero-sum trades, which is what HFT is designed to do.)
To the last part of your argument:
significantly change who is able to benefit from asymmetric information by inducing others to take the wrong end of zero-sum trades, which is what HFT is designed to do
This is simply untrue, nobody is being induced to do anything. Tactics like submitting orders you don't intend to let trade, to make other participants believe there is interest when there is not is illegal and enforcement is more aggressive than you might think.
To the remaining part, if you believe that HFT improves market efficiency to the extent that it exists, but don't think that the increasing competition provides additional value - OK. I'll work with that, but then if we can come around to a view that HFT is at worst neutral, why should it be curtailed? Simply because some think the people who practice it should spend their time elsewhere? I think that is a dangerous standard to enforce anywhere.
"Induce" may not have been the best word; I didn't mean to imply that manipulation of the sort you describe was going on.
What I meant was that the whole point of HFT is to get information about the market state, and act on it, a little bit faster than others; which means that when you make a bona fide offer based on your HFT algorithm's understanding of the market state, that understanding is based on information that the other party to the trade does not have, which means that the other party might accept a trade that, if they had the same information as you do, they would not accept. That is asymmetric information. There doesn't have to be any skulduggery going on; it's a natural consequence of what HFT algorithms are designed to do.
why should it be curtailed?
I'm not saying it should be curtailed; I'm saying that it's a shame that so much talent and effort goes into a zero-sum activity, and that that fact is a big part of why ordinary people distrust the financial system. In other words, HFT is giving other more beneficial financial activities a bad name. The right way to change that is for the financial system to police itself.
I think that is a dangerous standard to enforce anywhere.
I agree, and that's why I think the financial system needs to police itself, before it gets policed by others who don't agree with you and me that such a standard is dangerous to enforce.
Also on FX
If there is an API normally there is programmatic trading.
While I don't like the role of HFT in modern markets I find the technology that it drives very exciting.
They're (on average) worse than your average tech shop. But then again, people like google or facebook will never buy software from anyone, so if we're left with just them, you can never create a business writing software that caters for other software companies, not sure what is worse.
I have seen nobody ever challenge the value of liquidity in these discussions of HFT. Assuming this, the argument is that at some amount of latency the additional liquidity turns into a negative.
It seems more like the additional value is simply extremely small, but meaningful over an entire market.
What's the exact cutoff point for the value of additional liquidity turning negative, and will anyone agree on that arbitrary number?
Assuming HFT does turn into a negative at some point, wouldn't the same apply to optimizations in other markets like online advertising?
If it does apply elsewhere, we should also legislate away the practice of making online advertising more efficient to prevent tech giants like Google and Facebook from wasting billions of dollars and untold man-hours on these activities that are clearly worthless to society.
Your so wrong. Cash Equities account for notionally c. 50% volume at best the rest being made up from exchange traded derivatives. HFT, MM and Prop firms by speculating and posting contra liquidity lower transaction costs for other market participants. Society benefits from having a fluid and well greased market which reflects all known information.
Also get your Keynesian BS out of here.
In case of HFT, the argument is not of degree but existence. Military solves security problems of nation states, law firms communal issues of individual citizens, advertisements to help the product reach the intended audience. But what problem does HFT solve other than making a few people very rich.
The reason you hear so much about HFT is because the traditional firms it is displacing want you to.
Automated trading itself is still dangerous, in that many of these algorithms are black-boxes. They induce complexity and turbulence into the system. For example the minor bubble just before 2008 crash. May be automated trading/HFT was lucky that, the crash wasn't majorly their part. But it is a ticking time bomb.
Now with HFT it is all the more dangerous, because of the speed of interactions. If I make a bet in 50micro seconds, the space of possibilities explodes on you.
And how is it of enormous benefit to society, when it is concentrated in fewer hands. Again, these micro variations hardly say anything about the quality of the goods being produced than what other traders think about the goods. You are optimizing on a parameter, which does not model the problem you are solving. You might find a minima, but one that has nothing to do with making/distribution of these goods.
Secondly, anything that is profitable and legal is of benefit to society, unless it generates obviously negative externalities ( costs for others ) such as pollution
the reason is, our society is mainly built on the idea that everyone can pursue whatever they want and not what some central authority deems to be "useful"
"Anything that is profitable and legal is of benefit to society" is a little misleading. The definition of what is "legal" is an evolving thing. We know it is dangerous, in that can cause huge fluctuation in the financial control system without real benefit for the society as a whole except a very small percentage. I am not speaking of a central authority that decides it, but our common collective consciousness that should decide it. And again there is not one thing, which has no obvious negative externalities. But the question is whether the positives overcome the negatives. In this case, for the society as whole there is little benefit.
Exactly, when the very society is built on that idea that everyone can pursue whatever they want, the access to opportunities should be equal. HFTs are tilting the balance in one way (helping the rich getting richer) , can cause economic crashes, which hit the rich and poor alike. Now the aftereffects of crashes on the rich (except in case of exceptions) are not as tragic, as in case of the poor. For the poorer part of the society, it pulls the rug under them.
WTF are you talking about?
1. HFT's are full of math and computer science people, which are usually drawn from the middle and lower classes.
2. Before HFT's, the people on the floor were inbred... That's right. There was a lot of passing down the torch since it was the type of job that you can learn by doing it. Oh, yes, and to claim that they were selecting by some "meritocracity", they were selecting for "street smarts" Haha! That only gave them carte blanche to hire their relatives
They decide to create an even better new thing that has heavy capital requirements so they issue stock.
Even better new thing is not as great as we all thought, so stock goes out of favor as market participants channel capital to new and more promising tech companies.
Years down the road, MakeWorldABetterPlace.com, trading on old securities decides to enter the mobile market and creates revolutionary device. Market participants catch on and start channeling capital out from less promising securities and into the old and existing claims on assets of MakeWorldABetterPlace.com. Thousands of businesses and jobs are created by new privately held companies that create new things using MakeWorldABetterPlace.com's mobile operating platform.
So much talent focuses on the buying and selling of securities because it is the most social way to vote directly - by allocating capital. And the product of that capital flow is what is observed as the creation of new products that make the world better in a directly measurable way.
If you don't think liquidity is important, then with it goes the ability for countless tech startups to exit (through their acquisition by large corporations in exchange for claims on existing assets) and use the resulting capital to create subsequent startups, that perpetuate the cycle of creativity.
Virtually all trading volume today consists of buying and selling old securities -- essentially, legal claims on existing assets. The sale of new securities issued to finance the creation of new products and services -- for example, a company selling new shares via an IPO or issuing new bonds for investment in physical infrastructure -- represents only a minuscule portion of total trading volume.
I'm not sure having so many of our best and brightest minds going to Wall Street (and into high-frequency trading in particular) is a good thing, from a societal perspective.
--
PS. Whenever I read anything about high-frequency trading, I'm reminded of the following passage, written by John Maynard Keynes in 1936 -- 77 years ago (!): "Of the maxims of orthodox finance none, surely, is more anti-social than the fetish of liquidity, the doctrine that it is a positive virtue on the part of investment institutions to concentrate their resources upon the holding of 'liquid' securities. It forgets that there is no such thing as liquidity of investment for the community as a whole. The social object of skilled investment should be to defeat the dark forces of time and ignorance which envelop our future. The actual, private object of the most skilled investment to-day is 'to beat the gun,' as the Americans so well express it, to outwit the crowd, and to pass the bad, or depreciating, half-crown to the other fellow."
Source: http://ebooks.adelaide.edu.au/k/keynes/john_maynard/k44g/cha...
The more things change, the more they stay the same!