You speak as though there's an enormous gulf in producing social value between working at Facebook and working at Renaissance.

Speaking as a physicist who left the field, the vast majority of research physicists are not going to make a significant impact on advancing human knowledge. They're all too busy teaching bored undergrads, competing for underfunded grants, and politicking to get their name onto more papers.

I'd rather do math and make lots of money. Wouldn't you?

If you did do math, and you did make a lot of money - wouldn't it have only been possible because

> the vast majority of research physicists are....all too busy teaching bored undergrads,

?

I know more than a few dudes who work on Wall Street, most of whom graduated EE or CS from Carnegie Mellon/MIT/Harvard.

Most of them view it as a mercenary job, nothing more. They don't give a shit about finance itself, but they enjoy the money, they enjoy solving problems - especially ones that are seen as impossible - and obviously they also enjoy the deep technical aspects of the job as well. It scratches a lot of different itches and it's a very high prestige gig ("Wall Street", "Wolf of Wall Street", etc.) to boot.

Look at it another way: if they're out in the Valley, they're one of thousands of talented engineers, competing against gender disparity in a very competitive social arena. In NYC, they're young, brash hotshots working less hours for more money with all of the ego, bonuses, and cocaine that goes with it, in an infinitely more exciting city. Tell me which setup you'd rather have.

If you really are a talented machine learning specialist, you don't have too much to compete with in San Francisco. A lot of "ML engineers" and data science people in SF dont actually have the background or expertise to call themselves career machine learning specialists. They take free courses and started using tensorflow once it blew up. I know because I was one of those people (fwiw im going back to school to actually learn the fundamentals now).

Also, I know people in the situation you describe, and its a grind like every other bonus incentivized gig on wall street. People may tell you otherwise, but they're probably sugarcoating. I know a few people who burned out early and started their own companies, and a couple who are continuing the slog for huge bonus payouts. The no free lunch theorem holds true.

For me, I'll finish up this degree and see where things are headed. I'm definitely not set on SV or WS yet, but it'll probably be one or the other.

> very high prestige... Wolf of Wall Street... ego... cocaine

The lifestyle appeals to some, but not all, and Wall Street's "very high prestige" doesn't really exist outside NYC and its immediate neighbors. For most people, finance is associated with greed-fueled, drug-addled sociopaths who have to pay for sex and companionship, since that's what's portrayed in "Wall Street," "Wolf of Wall Street," the news, etc., and they don't associate that with prestige.

There are a lot of cautionary tales about Wall Street for a reason. Some in finance get out alive, others end up washed up or burnt out, barely maintaining employment, battling chronic depression due to years of abusing their brain chemistry. A million dollar bonus would be nice, but the other stuff I can do without.

Someone else mentioned pandas and AQR; I'll also mention BeakerX (Beaker Notebook) which was developed by Two Sigma: https://github.com/twosigma/beakerx

You can read more about it here, too: http://beakernotebook.com

Jane Street is also pretty awesome - they open sourced their in-house standard library for OCaml, called Base: https://github.com/janestreet/base (they use a lot of OCaml).

And to be clear, although Base is a more recent alternative, Core: https://github.com/janestreet/core has been the standard library for Ocaml for a while.

There's nothing handwavy about the liquidity argument. Improving liquidity reduces the cost of trading for almost everyone in the market and thus makes almost everyone just a little wealthier. That means ordinary people, like schoolteachers and custodians, retire with a bit more money, and are a bit more comfortable; it means lots of people can donate just a bit more money to charitable causes without crossing whatever financial safety line they have, &c.

It's hard to make the same kind of argument about adtech.

No one disputes the quantitative benefits of increasing liquidity. But I've yet to see it quantified in a mathematically rigorous way. Are the speculators and active traders actually generating more value for society as a whole then they're syphoning off? (That's an honest question, I'd really like to know.)

I think that, in terms of social value, the point of diminishing returns for market making and liquidity has long since passed, especially wrt social value for Avg. Joes & Janes.

I def. don't hold Facebook etc in a much higher regard here, but if we're talking about brain drain from socially useful fields, I don't think anyone can credibly argue that finance (esp. HFT et al) provides any meaningful social value.

In fact I'll go one further: by sucking up the best and brightest, high finance is doing a net harm to more socially useful sectors.

I'm not being prescriptive wrt what choices people should make, btw, just speaking in terms of social good.

This is somewhat self correcting.

HFT firms are make money by taking a spread, which is effectively the cost of making a single trade. As they compete and get better and better at there jobs, this spread well be reduced, the amount of money available to pay for the best and brightest programmers will come down, and they'll suck fewer of them away from other parts of the economy.

The point being, the further they go past this 'point of diminishing returns', the less money they can possibly make. To the extent that HFT is a zero sum game, there will always be a limit on how much money they can spend.

(I believe but cannot prove that this limiting effect has already started, HFT firms are consolidating and making thinner margins than they used to)

That's a good point, but would that happen quickly enough to undo whatever brain drain problems the industry creates?

Nearly all hedge funds still take 2/20 for example, and their returns have been questionable for a decade, so idk we have reason to have faith in high finance's ability to self correct.

Every time someone brings up social good, I keep wanting to know what that means. Can we have a truly objective definition of 'social good'? Or is it entirely dependent on the individual person's definition of what that means to them.

I agree that it's not a sufficiently precise term, but I think it's obtuse to suggest it's so ambiguous as to be deceptive or obstructive to discussion.

How's about: an act that is socially valuable has benefits, immediate or otherwise, realized by parties that aren't directly involved in the transaction.

Eg, a blackjack dealer's labor probably only matters to her players and her boss, but a cancer researcher impacts well beyond that. Both can be lucrative, but one is clearly more socially valuable.

i agree with you and i think it is a good point but just one doubt i have that liquidity argument is correct assuming that this doesn't effect the fundamental pricing and cause bubbles (for ex.[oversimplified one] if lot of people followed trend following - it keeps driving bubbles in the long run - although i know counter argument is markets actually become efficient with more trading, but it is hard for me to buy into that 100% - i hope i'm wrong though). Also I think a decent gauge would be how much open-source has been contributed back and how much of it is valuable. Other things are hard to quantify like the example you gave, but I can't prove that you're wrong either.

Also when you develop something like physics, once proved - it always works with very good precision(reason we can launch satellites accurately). Or let's say a software or database that is supposed to work a certain way. What's being done at quant funds - I can't think of something on those lines that could be useful. Lot of these things don't always work because you're still speculating with large inaccuracy but a slight edge statistically.

Some interesting links:

http://fortune.com/2016/05/11/warren-buffett-hedge-fund-bet/

http://www.marketwatch.com/story/why-99-of-trading-is-pointl...

I'm not going to comment on the liquidity argument (it's a dead horse in my opinion), but another commenter and I mentioned a few examples of open source contributed by "quant" funds.

Not everyone at Google works on adtech. There are many people working on browsers that have sped up and innovated the web for hundreds of millions of users, or operating systems that are the foundations of many people's lives.

I didn't say Google, I said adtech.

Also, one could say the value that Android has added as an open-source alternative to Ios has a value to society in the tens of billions.

Ordinary people eh? Extraordinary are you?

All I see Wall St and Silicon Valley doing is using new found speed and scale for their own benefit while concocting self deluding narratives about how this benefits everyone. Nothing extraordinary about that.

Thomas isn't being condescending (at least not intentionally) when he uses the term, "ordinary." He would also fall into "ordinary" under his own definition, as he's not in that industry.

His point is that, contrary to the narrative of the "little guy" getting screwed by Wall St, they are actually helped by the liquidity HFT market makers contribute.

Everyone is unsophisticated at something. It's not an insult.

Unfortunately, there is such a thing as too much liquidity as it can cause turbulence. Extending the metaphor, a viscous fluid might be more stable.

pandas was first developed at AQR Capital.

http://pandas.pydata.org/community.html

Cool, thanks, that's what I was hoping to learn about.

none, their goal is to optimize profits. They haven't gotten to a point where investing in cutting edge tech only research is seen by them as a value add. Also for such projects they stay away from open source collaboration with outside world - so that would be really slow to develop something useful. Hope that will change in future.(very few exceptions - as someone commented pandas - pandas creator at two sigma actually is pushing for open-source, i think it is great but i hope it is not just a gimmick to attract better talent).

To be honest with you, I consider most examples of companies releasing open source software to be commoditizing their complements and streamlining recruiting, not altruism.

The result can be altruistic, I guess, but my point is that judging intentions with regards to open source on the part of large companies is a fool's errand in my opinion.

good point, what do you think about tensorflow, openai gym, python, apache foundation, linux foundation? you gotta have some love for open source if you're a developer. i might not be as well read as you but i would like to be corrected if my thoughts are incoherent. also i don't think it is about altruism so i agree with you, there's mutual incentive to open source things like tensorflow i.e. to speed up development.

I use Tensorflow, but it's a fantastic example of my point. Google open sourced Tensorflow because the real strategic advantage in machine learning is data and talent. By commoditizing the software used for machine learning, they broaden the market for their expensive, high margin data, and lower the barrier for talent to enter the industry which they can hire.

The likeliness for ending up in a tenure-track research position is much much lower than it is to land a good job working at wall street.