Corporations already have legal personhood and act in their own interests. It's going to be much easier to automate and formalise business decision making than to develop a true general intelligence with a full spectrum of human characteristics.
This may sound like science fiction, but as competence increases shareholders - who typically are only passingly interested in moral issues - are likely to demand the increased returns an AI CEO can bring.
But AIs don't, and corporate officers must be persons (natural persons, even -- a corporation can't be a corporate officer.)
That in itself is not so bad, as extremely long-timeframe constraints (say, >50 years) upon such an algorithm could conceivably be consonant with current decision-making behavior that externalizes many input costs (employee overtime, environmental damage, etc.). Running the algorithm to pay out in very short timeframes (a month to a year) due to most CEOs' anticipated short tenure is what seems to cause undesirable optimizations.
A tiger doesn't need to be self aware or have intent to be dangerous.
The paperclip maximizer generally refers to a AGI with a value system that is not aligned with humans. An AGI smart enough to achieve its goals (making paperclips) so efficiently that it becomes a threat to humans through sheer resource consumption.
So it's not a good example of dumb-tiger-AIs occasionally becoming a threat to humans, which still on average are able to outcompete a tiger with ease.
The "system" does not need to have intent or be even closely aware to be dangerous to humans.
And so the article sets up a false premise if the quoted conclusion is to be the base of judging whether it's going to be a threat to us or not.
I am thinking about systems set up to protect us can end up hurting us
http://assets.motherjones.com/media/2013/05/LakeMichigan-Fin...
5 years ago it was thought to take decades to beat a human in Go.
Just 10 years ago self driving cars was something you joked about.
We consistently overestimate progress in the short run and underestimate in the long.
You tried to use one expert as a refutation of my claim that there was a general consensus with how long it would take for computers to beat a human in Go.
A simple search on google will provide you with plenty of writing backing that sentiment up.
Asking any of your friends who took AI classes back then would confirm the same.
Given your point was a link with an example of one person who believed computers would beat GO you were doing more than that.
Of course there were people who believed that Go would be able to beat humans. Just as today there are people who believe that AI can be a treat.
But it wasn't a majority of people who believed Go would be able just as it isn't a majority who believes AI can be a threat.
I.e. just because the majority believe something doesn't mean it will be so (or vice versa)
And not media reports from present-day that just repeat this meme that almost everyone believed Go wouldn't happen for decades.
A highly prominent expert opinion but the general consensus was that it would take a long time. Ask anyone who went to AI class back then.
Here is another expert
"In May of 2014, Wired published a feature titled, “The Mystery of Go, the Ancient Game That Computers Still Can’t Win,” where computer scientist Rémi Coulom estimated we were a decade away from having a computer beat a professional Go player. (To his credit, he also said he didn’t like making predictions.)"
http://www.wired.com/2014/05/the-world-of-computer-go/
You also find highly prominent expert opinions that AI is going to be dangerous and experts who don't believe it. Most people don't believe it, most people believe robots wont take jobs either.
And no I don't need to provide you with anything since you have only problem my point that most people didn't believe it would happen which is why you didn't link to anything saying that most believed it would happen.
I think it may be rare that you see consensus on those sorts of "it's imminent, someone just has to do the work" problems because it requires simultaneous knowledge of multiple developments, and knowledge doesn't always disseminate as fast as it takes one group to just do the work. Now I'm remembering this related maxim: http://lesswrong.com/lw/kj/no_one_knows_what_science_doesnt_...
Reply to below: You're the one asserting that experts thought Go wouldn't be dominated by computers for a long time. The burden of proof lies on you. "Some experts thought it would happen by now, some didn't, there was no consensus" doesn't have quite the ring to it!
I gave you both an expert and and an article claiming backing my claim up.
Find me on single article claiming that it's common knowledge that Go would beat a human soon back then and you have me.
Until then I am pretty confident the main consensus was as I claimed and which is my main point.
We mostly underestimate how fast this is moving and it's not only laymen who get it wrong, many experts do to.
You have provided on example, ONE of someone who believed it would happen.
I have provided one expert plus articles saying it wouldn't happen, plus you can google and find plenty of articles that said we wouldn't get it for a long time.
You cannot find a single example of articles which are claiming that it's was a general consensus we would beat Go.
And so you are the one coming up short not me. My claim is not controversial neither have you shown it is.
However, points to argonaut for doing his best Dijkstra impersonation.
'I don't know how many of you have ever met Dijkstra, but you probably know that arrogance in computer science is measured in nano-Dijkstras.' - Alan Kay
That changed on the second day of the 2005 DARPA Grand Challenge. Suddenly, there were lots of self-driving cars running around. The sudden change in the attitude of the reporters there was remarkable.
https://en.wikipedia.org/wiki/DARPA_Grand_Challenge
Several vehicles finished the 2005 course. The one that finished first won a $2 million prize.
you can at least ride in one (if youve got bus fare)
You can't overestimate the economic pressures on progress as well.
Remember in 2007 when everybody thumbed their noses at hybrid and electric vehicles in the US? Ford was still pumping out record numbers of their behemoth Excursion model.
Then the economy crashed, and people suddenly needed a fuel efficient car and then they all traded in their SUV's for what? Toyota Prius' which were an after thought a few years prior - in the span of 18 months, Toyota couldn't keep them on the lot.
I can see one or more catastrophic disasters where there is a sudden need for AI to rescue the human race in some capacity. Think nuclear war, environmental disaster, biological catastrophe, etc.
https://en.wikipedia.org/wiki/AI_winter
Too much opportunistic thinkinking caused crazy hype cycles.
I guess compound interest could be considered indefinitely exponential, but you eventually reach a barrier in what's insured and it is a relatively small exponent. Also, is it still savings if you never spend it? I wonder what is the longest continuous account in banking that has never been touched.
Anyway, that is a tangent and a somewhat artificial scenario. Can you name a naturally occurring scenario? I would accept technology if you could show that it isn't going to level out like all other natural phenomenon.
Who says it needs to be exponential forever?
(Humans, fortunately, are starting to get better about this.)
Also, you want "logarithmic," not "logistic."
Don't think so - https://en.wikipedia.org/wiki/Logistic_function
The 'S' curve of logistic growth looks exponential for a while, which is why the question arises. By contrast, no one mistakes logarithmic growth for exponential growth for very long.
That said, it's typical for even experts to underestimate progress in rapidly advancing fields. No one predicted AI would be so good by now, say 5-10 years ago. Now computers are beating Go and rivaling human vision.
Giving this the headline of "One Hundred Year Study.." was confusing. That's the name of the ongoing effort to do this kind of analysis, but the paper is named "Artificial Intelligence and Life in 2030"