Holy anti-intellectualism, Batman!
If you describe machine learning as "data juggling bullshit", this is very strong evidence that you simply don't understand what it is. This is an indictment of you, not of machine learning. Machine learning would more accurately be called "applied computational statistics" in 99% of cases.
What makes you think that the people using applied statistics to make money are "unwise"? Based on your tone, I would guess that it's because what they're doing doesn't agree with your folk definition of "an honest day's work" or something like that. This isn't really a good criticism; it just means you don't see the utility of what they're doing, which requires some degree of abstract thinking about the market.
Particularly the problem of this website, is it's huge focus on anonymity. You don't need to expose your code or methods, or even your name. It's an ongoing competition so it encourages secrecy so you can make more money next week.
This means even if someone does invent a new super ML method, they have every incentive to not share it and keep it a secret. So this website could actually have a net negative impact on the world.
Why spend 80 hours a week chasing grant money when you can spend 60 moving numbers around?
In fact I'd say this job is quite uniquely, extremely, only about the money. Even when compared to other kinds sweet/lucrative, perhaps "stupid lucky" jobs you may find, that may require 60 hours mindlessly pushing papers, yet paid extremely well. I'd be hard-pressed to come up with any hypothetical kind of job could be more sharply, singularly focussed on the "do extremely-intelligent-monkey-dance, receive ample bio-survival-tickets" to the exclusion of any other meaning or fulfilment than this Encrypted Machine Lottery/Learning/Sudoku.
Second, I could be wrong (or too optimistic) but I'd hope that most "very intelligent" people demand more nourishing reward than just money to regularly put in more than 40h/wk. You can do this for research and advancement of science, or maybe because one considers the work itself a net positive to society, or maybe you chose to sacrifice those hours of your life to support loved ones, maybe there's some deep personal reward in it, or maybe it's temporary and you're saving for something rewarding. But to pretend just money is enough to waste your life on, hey it's a choice, but I'm going to have to see you turn in your "very intelligent"-card.
On the other hand, there will be exceptions. Some people don't care, just want to maximize $$$ for the least amount of effort/hours of one's life, regardless of side-effects. This website does not fill a niche for these people. If you are extremely intelligent, just want money and don't care about moral aspects, there's always been plenty "business opportunities" to fulfil these particular needs.
Finally, if one would argue, but this job isn't quite explicitly "badwrong" like those others. I'd suggest to think it through: On the one hand, a (possibly) well-paying job that is designed from the bottom up to have no way of determining whether it has a net positive or negative external effect besides the money it pays you. On the other hand, literally anything else you could do with your time.
I can see people trying this for a short time, as a funny puzzle, at most.
[0] I don't mean this one, but because I still think it's funny, I'll point out the way-too-easy retort here: this says something (not terribly good) about you. (j/k)
Actually, it does. Increasing liquidity, communicating price information, more accurately accounting for predictable future changes in price, and absorbing risk allow for increased agricultural and industrial production. Like I said, you have to think through a few layers of abstraction.
A nice rule of thumb is that if someone is getting paid to do something you personally think is useless, it's either a government job or you just don't get what they're doing, but the market does.
But the opportunity cost is what I'm getting at. So many very intelligent people could be producing immense value in other areas, but instead drained away doing this garbage. And they are incentivized to keep anything useful they invent a secret.
http://www.investmentreview.com/files/2010/07/The-value-of-l...
https://fp7.portals.mbs.ac.uk/Portals/59/docs/Finger,%20Mark...
http://people.stern.nyu.edu/adamodar/pdfiles/papers/liquidit...
https://www.macquarie.com.au/dafiles/Internet/mgl/au/mfg/mim...
https://en.m.wikipedia.org/wiki/Liquidity_crisis
The stock market is absolutely connected to "real value" (which doesn't make sense as a concept in the way you're trying to use it). Besides allowing companies to raise capital for future endeavors, it also allows the market to communicate price information, which is critical for reducing the risk associated with a given transaction.
> at best any benefits just benefit rich investors buying stocks.
It benefits everyone; investors risk their capital by selling it to companies in exchange for partial ownership, which allows the company to grow, and if the investment pays off the person who risked their money is rewarded. Without the stock market, companies would have to get loans for growth, which is risky and too expensive to be practical.
> It certainly doesn't increase production.
Yes, it very much does, for the reasons I outlined above. This is like Econ 101; why are you commenting so strongly when you clearly do not have any background here?
> but instead drained away doing this garbage.
A small efficiency increase expressed over 100 trillion dollars of trades every year is a huge productivity increase for society.
Look, for instance, at high frequency trading. Traders spend millions of dollars trying to get signals across the earth a millisecond faster than the competitors. Does it actually benefit anyone that information travels a millisecond faster? Who benefits from the high frequency traders at all? Yet they eat up millions, maybe even billions, of the economy's resources doing this nonsense.
Look at these trader spending millions of dollars to fly drones over oil tankers or parking lots, to get slightly more accurate information faster than anyone else. Look at these hedge funds spending tons of money on bribes to people in companies to get insider information before the public does. These things clearly provide zero benefit to the world, it's just a massive waste of resources.
This hedge fund is a bit different, in that they seem to be actually doing fundamentals and longer term bets. That's not so bad, but it's still pretty disconnected from any real world benefits. At best they predict a company will increase in value and buy it's stock. How does that benefit anyone? It's almost a zero sum game. The people that buy the stock that week lose, because the price is higher than it otherwise would have been. The people that sold the stock lose, because it was really worth more than they actually sold it for. Any money the hedge fund makes necessarily comes from someone else losing that money.
The stock market itself is not really connected to the real world. Sure sometimes companies sell stock to raise capital. But most stocks being sold and bought are not by the companies themselves. It's a huge indirect chain of traders and investors and speculators, trading these stocks many, many years after the company sold them. And the company itself would still be able to sell stocks if there were fewer traders. At worst the prices would be slightly less accurate.
Price information of stocks is also a horribly inefficient way of getting this information. These companies are often involved in many separate projects. So even if someone has a model that can exactly predict the success of products, it's very difficult to use that information. You also need to determine how much the company is worth given all of it's projects and businesses and property it owns. It doesn't directly give that information to the company itself - they see their stock went down, but they may have no idea why. Not to mention Keynesian beauty contest problems, where you have to predict not what the company is worth, but what other people think it will be worth, and how much they know currently, and what the interest rate is, etc.
But you didn't address my main concern, which is the bad incentives of secrecy, and the opportunity cost of employing smart people doing this, instead of doing something else. Even if it does produce some value, it doesn't mean it's worth the cost.
The stock market is one of the mechanisms that allows the market to be right by communicating price information.
Because it is an incredible waste of talent.
I know this is in contradiction with all the 'values' that have been drilled into our minds since we were born, but it's about time we wake up and reorder our priorities.
>Based on your tone, I would guess that it's because what they're doing doesn't agree with your folk definition of "an honest day's work" or something like that.
If you look at the state of the biosphere / atmosphere / oceans - the data - and if you have children - then it should be quite obvious.
Also there are the socio-economic challenges that the world faces right now - in fact it is unclear if we're going to make it to the next century as a species.
It really don't matter how much "money" you have in your account when your city sinks under the ocean...
The fact that these people chose to spend their time doing numerai probably means that they think that's the best use of their time.
The only way we're going to wipe ourselves out is with nukes, nanotech, or engineered viruses. Global warming is not an insurmountable concern. It will be expensive for coastal cities, though.
There's no reason we can't focus on multiple things. Global warming isn't so pressing that we need to dedicate all intellectual output to it. Every 10 or 20 years, the environmental alarmists come up with something new to obsess over (I think it was landfills previously), and they're doing it now with global warming.
You just move. That's what money enables. No city is going to sink so rapidly people just drown.
I believe he's suggesting that there are a lot of other areas where their knowledge and skills could be applied which would create some societal benefit as a byproduct.
It does both, the latter by more efficiently communicating price information and incorporating future events into asset prices. More accurate AI traders absorb risk, which is good for the production economy.
They are in effect crowd sourcing curve fitting because the number of possible models for this maths problem is so large.
Think of crowd sourcing here as a pruning heauristic in a search problem.
Ironically, it's also not entirely a math problem - the underlying system doesn't exactly follow the expected rules of math.
This is not a pure math problem. Eventually the outcome of all these models and predictions affects the stock prices and - if it becomes as successful as you hope - the economy as a whole. And the physical world: people, animals, plants, pollution, CO2 and so on.
I would much rather see work in that area - than this data juggling deep learning bullshit which results in profits being paid out to a bunch of intelligent, greedy and unwise people.
I just hope that when the intelligent people finally become wise, it won't be too late.