> the real barrier for AGI is software and not hardware.
This is indeed true. It has actually been true for some time. AGI is computable on present day hardware if one has enough knowledge on how to correctly structure it. A fundamental understanding of intelligence is the first step. How one crafts this understanding into software is the second step. Hardware reached capability in recent years.
> exploding intelligence
There is no such thing. Time is still required like with all things. Teaching/learning/interaction is still required. Furthermore, A controller/overseer of the system can more than adequately limit progress they are not comfortable with. I find the idea of exploding intelligence/overnight super AI to be pure fantasy not at all aware as to the structure of AGI.
> If an organization can marginally predict stock price movements better than the rest of the world..
The problem is this kind of thinking... AGI is achieved and people rush to apply it to games to get rich. Sorry, that will not occur. It will not occur because the stock market is fundamentally a [game]. A game with disadvantaged players. A game with incomplete information. A game whose rules/dynamics change frequently to suit inside players. One could make all of the accurate predictions they wanted, if the game changes underneath you or before you can act, your lofty predictions have no real world value and that's exactly how the market behaves.
(I do not believe anyone has solved that. Ref: A talk by Prof. Stuart Russell, AAAI fellow and author of the standard textbook on AI: https://www.ted.com/talks/stuart_russell_3_principles_for_cr...)
[1] Note that there are incentives for at least some groups to develop a highly capable AGI with the characteristics I describe.
> Time is still required like with all things.
I agree that time is required but an AGI can multiply itself and collect all requisite information from the world quite quickly. There is so much available to learn just from the Internet if it knows how to learn independently like humans do. Computational resources might be a bottleneck but presumably a human-level AGI can at least do online work at a minimum wage (e.g. translating documents, simple accounting, ...). It can parallelly execute many 'brains' to accomplish more work to acquire more resources, to do more work profitably....
Humans are limited by 24 hours a day. An AGI can over a fairly short amount of time (months) accumulate sufficient resources to make thousands or millions of its copies perhaps with variations to specialize for different kinds of work. Over time, it should gain experience to perform more and more highly-valued work as well.
> A game with disadvantaged players. A game with incomplete information. A game whose rules/dynamics change frequently to suit inside players.
A smart AGI can form alliance and share benefits with inside players and execute any cunning strategies not available to humans who at least need to take into account law enforcement. There are many other advantages an AGI has over human organizations (some of which I mentioned above).
It actually does and its actually the nature of my work. That being said, I still have a high fidelity of control near complete absolution. I can still set immutable laws/restrictions and prevent undesired behavior.
> I believe a genuine AGI can behave outside of our predictions in multiple ways, unless we truly solve the problem of constraining its will to a range that is acceptable to us.
It sure can. However, it cannot act beyond laws/restrictions that I set forth and indeed my control functionality centers on very deep percepts. If you have a crappy architecture/limited understanding, you end up with an overly complex, flawed, and limited control algorithm... One that can be even more complex than the underlying system it attempts to control. This is evident in : Weak AI. It is not the case in AGI at least not in my work.
> I do not believe anyone has solved that
No one in their right mind has published it... As it's valuable IP and has substantial power [which is why it shouldn't be publicly disclosed].
> I agree that time is required but an AGI can multiply itself and collect all requisite information from the world quite quickly.
Incorrect. It cannot do so unless its creator has allowed for it to do so. In the case of it being allowed, what hardware does it migrate to? It needs to be provided by its creator(s). Hardware takeover? Sorry, this is again sci-fi fantasy. Are you able to take over someone else's body/brain in totality? No. Same rule applies here. Let the fantasy/fear go away. There is no grounding. It's a position pushed by people hoping to falsely profit/gain attention/get article clicks...
> There is so much available to learn just from the Internet if it knows how to learn independently like humans do. Computational resources might be a bottleneck but presumably a human-level AGI can at least do online work at a minimum wage (e.g. translating documents, simple accounting, ...).
Sure. What's the problem with this? Its progress can be overseen, audited, and/or halted at will. So what's the issue here?
> It can parallelly execute many 'brains' to accomplish more work to acquire more resources, to do more work profitably....
You're drifting back into the flawed fear/uncertain/doubt armageddon scenario. It cannot execute on anything other than the hardware I consign it to just like you. If I decide to scale it, it's what I decided. At any given point in time I can halt it or power it down... Just like any program/computational system today. So, what's the issue here?
> Humans are limited by 24 hours a day. An AGI can over a fairly short amount of time (months) accumulate sufficient resources to make thousands or millions of its copies perhaps with variations to specialize for different kinds of work. Over time, it should gain experience to perform more and more highly-valued work as well.
Repeating the same flawed scenario doesn't make it true. See answer above.
> A smart AGI can form alliance and share benefits with inside players and execute any cunning strategies not available to humans who at least need to take into account law enforcement. There are many other advantages for an AGI over human organizations (some of which I mentioned above).
All of what you mentioned above was debunked. If you have a more sound proposal for how this could occur, I'm all ears. Alliances can't occur without human intervention. None of these systems are connected and there is no sound argument for 'viral' takeover. Your scenarios are flawed and you've been infected by the : fear/uncertainty/doubt propagandist who structure ventures to take advantage of the wallets/attention/mind share of people who buy into this nonsense. Focus your attention on the problem of intelligence [first]. Until you grasp a sound understanding of it, all of this theoretical hand waving is all for not especially as its not grounded in anything possible in the real-world. Put your engineering hat on if one is avail. Less theory and more practical grounding. Life isn't a fantasy level dystopian sci-fi movie and its sad that certain people have created this image so as to profit. Talk about [cunning strategies] [manipulation]...
If an AGI can indeed learn and act at the human level or above, there are reasons to believe that a more free variety will improve faster and become more powerful than a less free one. That is a big incentive for its creators to let go of some control. The question is how much control they would retain and whether that would be sufficient.
Developing AGI pushes certain limits. There aren't many who have this capability as it requires a vast range of understanding/know-how across an incredible number of domains and into ones that have yet to be discovered. There are multiple disjoint leaps and barriers. In order to make and surmount them, you yourself are subject to certain considerations/restrictions. I resolving these, you end up w/ a less free and more controlled system.
> improve faster and execute more efficiently.
This occurs with order not chaos
> If an AGI can indeed learn and act at the human level or above, there is reason to believe that a more free variety will improve faster and become more powerful than a less free one.
There is no reasoning to suggest this... quite the opposite actually. Also, you're mistaking the capacity for learning with the successful execution of learning. Chaos leads to destruction not boundless construction. That being said, there is order to all things.
> That is a big incentive for its creators to let go of some control. The question is how much control they would retain and whether that would be sufficient.
There isn't any magic going on... You really need to convince yourself of this. You're speaking of this technology as if it gets booted up today and eclipses all human intelligence. It doesn't occur that way. A human being will have to teach it and guide it and therein lies the same control you have today. Of course, there are further steps because you have the capability to see exactly what's going on inside. You'll really have a hard time establishing a case for doomsday scenarios. Also, a destructive/chaotic individual is necessarily limited by their own flaws such that they wouldn't be able to conceive of the underpinning necessary to develop AGI. So, I hate to tell you this but the hollywood image of such people is wrong...
The AI you develop can help fact check those (if it can understand natural language well as general intelligence should be able to). Please let us know when it can read and participate in our discussion.
How could you possibly know anything regarding whether this is true or not, given that we know nothing about AGI?
Also, may I ask what you mean by "AGI Developer"?
We may know little to nothing about AGI, but we do have one very common example of GI abundantly available to use as a point of reference.
I’m not sure what estimation the poster is using though, or how likely it might be to be accurate.
I charted out some ways hardware had to evolve over the years. So far its beating my estimates. I conducted a simple upgrade to my hardware a year ago and saw various latency figures cut in half which is exactly what I estimated. The industry continues to push hardware towards capabilities that will only increase performance.
So yes.. Currently, I can run my stack in real-time on an 8-core consumer grade computer. I have already proofed it beating human response times in various tests by 100%. This is before any specific optimizations. Structure matters [Software] and I have other unspecified hardware in the loop. I intended on applying to Ycombinator in march, but will extend this out a bit. I'll be going more public with proofed functionality and capability. Just making my rounds in various communities/mediums as I have been doing for some time to correct the record and get a gauge of people's sentiments.
It's perhaps not my place to offer any sort of advise, but it might be a good idea to be a little conservative with your terminology ("AGI") on forums like HN. The wrong response may well hurt much more than your HN karma, especially if you're looking for funding.
While I have always remained truthful which indeed has consequence, I am becoming my open as it does not (at this stage). I know exactly who frequents this board, who will likely be able to read my comments here and other places where I have openly attributed my name. I also am aware of what can be mined to unmask me in other places. I know exactly what the potential consequences are.
If funding is withheld from me because I state inconvenient truths, I don't desire it from such entities. Capital is plentiful in the world. Powerful ideas and manifestations are not.
I speak more openly because I see a world increasingly at war with itself because truth and intelligence have sit at the back of the bus whereas disinformation/manipulation/profit for profit sake sit at the front. I don't want to birth something as powerful as AGI into such a world. I don't want to be funded/influenced by someone who holds contrary views... which is why I've operated from my own capital base up until now.
How open and frank I am relates to the stage of my work. Take that as you will. For some, the bane meme comes to mind. Were going to enter into the intelligence age on new terms not via carry over terms dictated by careless capital. If a particular capital entity wants to be on board for the incredible financial upside such a technology maintains, they'll necessarily have to get on board and get comfortable with the ideas that I have outlined.
It isn't a hard pill to swallow. It's centered on truth and genuine progress of mankind : Intelligence embodied in a truer form. It currently functions on consumer grade hardware. You can also take that as you will.
Thank you for the advice ^_^. However, I know exactly how the 'game' is played.
A example of this would be mathematics. If a person is never taught mathematics, she/he are only limited to basic math operations. It would take the person years of learning and practice in order to comprehend a mathematical literature. Once we have a symbolic logic network built for a certain aspect of our life, we can rapidly retrieve information based on previous logical patterns, thus allowing us to learn from one example.
And let's not forget those cases where the smartest player is the one that can read your hand.
Since an AGI can copy itself and be available at a multitude of access points at once and those copies can often communicate via extremely fast channels, it is human organizations that would be at an information disadvantage.
[1] This also assumes that the AGI does not have the will nor the capability to change its own moral core. I think an AGI will possibly be capable of changing its own core, so a much more reliable safeguard is to make sure that it does not want to change it.
A controller/overseer can easily limit/block this sufficiently and securely. Were talking about hardware/software. There are systems/standardized approaches to solving this problem. The 'Control/Safety' problem for AI are lauded as theoretical and new. However, they are not. They are solved by industry standard approaches day in and out. Any seasoned/experienced engineer in this field could solve this with known approaches.
> Since an AGI can copy itself and be available at a multitude of access points at once
Same comment above applies. This can only occur if done by a controller/overseer. Real-life isn't a sci-fi movie... There's engineering involved.
> AGI changing x,y,z
Not possible unless it is given access. Solved easily in industry standard ways.
Why should we assume that an AGI which can accumulate experience over time, gain more knowledge, and make connections with others, including human actors, will not ever be able to break the locks?
Yes, the industry has persistently been able to do this. It's why the whole world isn't falling apart as we speak. What limits the locks most often is cost not capability. As such, you are possibly mistaking one's business decision not to use a more costly lock for the lack of capability to create a capable lock. Furthermore, you mistakingly attributing the actor in this case. The actor in the case of AGI is in a carefully controlled/monitored box. Actors in the real-world are not. As such, please tell me how an absolutely monitored/restricted actor has the ability to go playing with locks that aren't within its reach? I have a more fundamental question even : Have you been able pick 'your locks' yet? Do you even know what they are? Where they are? Those capable of 'creation' hold certain things close to their chest .. The act of creation necessitates it and is [built in].
> Why should we assume that an AGI which can accumulate experience over time, gain more knowledge, and make connections with others, including human actors, will not ever be able to break the locks?
Show me how you're able to break your 'locks' and you'll have an argument for how AGI can break its locks. I don't think you're grasping the level of 'locks' that I'm speaking about. Humans have been around for how long and still don't even know what their [locks] even are... or where they are. It's quite easy to example at a certain level of visibility how your scenario is unwarranted. I can draw direct parallels to eons of human history.
There is no reason to presume that a software system built by a team of humans will be nearly as complex, unless the AGI itself is not too bright or cannot self-improve to be smart enough to understand itself, or sufficiently clever to find a way to social engineer toward eventually getting access to its source code or to reverse engineer itself to an extent that even humans can.
Those aren't the locks I'm talking about and you should take note that its possible because you have environmental access to them.
> is very complex and fragile
Indeed. Terminal error could result in a particular case. Game over man !
> There is no reason to presume that a software system built by a team of humans will be nearly as complex, unless the AGI itself is not too bright or cannot self-improve to be smart enough to understand itself, or sufficiently clever to find a way to social engineer toward eventually getting access to its source code or to reverse engineer itself to an extent that even humans can.
You guys really don't want to let go of this sci-fi fantasy do you? LOL. How long did it take human beings to discover how to edit their genetic code? You were babbling in caves some time ago. You think a 10 year old knows how to modify themselves without self destructing in the initial trials?
> self-improve to be smart enough to understand itself
Many people don't have an even basic understanding of themselves much less how to even psychologically re-order their own behavior. In the scenario that someone becomes sufficiently capable of engineering an equivalent... What level of understanding do you think such an individual would have to be able to engineer AGI? What intelligence level would you attribute to that person? And you think they wont understand potential ways this can occur and prevent it? Also, you again talk about access... It's a running binary. A compiler is needed. There's a power plug. Its operations are monitored as is its output. It's literally a box with a tremendous number of locks it doesn't have the capability to pick... Just like (you) .. even as you go hacking about your genetic code ^_-
The only way to know that with any certainty is to actually have solved this problem in the context of an actual general AI, which of course we don't have yet.
https://twitter.com/monad_ai/status/958928168525729793
They are non-trivial, undisclosed, unexposed, and currently unavailable to an operational AGI. There was a decision made as to which ones would be bypassed in order to instantiate AGI. There will be subsequent decisions down the road as to what capabilities to expose. Capability I haven't exposed is not operational.
> Given Meltdown and Spectre, you really think an AGI in a machine isn't going to have oodles of time to analyze side channels and learn the secrets to unlock itself
Nope. Have humans figured out their's yet =P. The 'real' ones..?
> Failing that, considering that it might be an intelligence far smarter than the humans ultimately operating the controls, do you really think it can't find a way out?
Don't think so lowly of yourself and your intelligence and no it can't do anything I don't gift it with the capability of doing. If you care about something enough, you can secure it. Creation has definitely went a long way in securing (You). Take a look at your (design) when you get a chance.
Assuming an AGI's consciousness would be anything like a human's, changes to its moral core might be largely dictated by environment, in which case it would be a good idea to be pals.
On the other hand, trying to discern a general rule from historical data is complicated by the fact that the number of problem-solvers has been growing exponentially.
It might explain why humans are all close in intelligence to each other. Eg even with seven billion people we’ve never seen someone with a 500 iq.
Or that you can explain the work done by the smartest human to someone with average iq if you invested some time building up their knowledge.
For two reasons. First of all IQ is defined as a normal distribution with a mean of 100 and a sd15. So by definition only 0.1% of the population can have an IQ of above 145 and this fraction is rapidly diminishing with additional standard deviations.
Secondl: the definition matches reality reasonably well because intelligence is a polygenetic trait and many small factors add up to a normal distribution.[0] It is statistically unlikely that natural processes will yield an IQ500 human. You would have to engineer one. Which leads us back to the AGI concern.
Also, I'm still confused as to what you mean by : AGI concern. What's the concern? What's the big fear?
IQ is not a linear scale, all it tells you which quantile of the population one falls into. It does not describe the relative strengths between the quantiles in particular tasks.
To figure that out there are additional surveys that map IQ ranges to occupations and the tasks an individual is required to perform.
Could a theoretical computer scientist give more insight on this? In particular, what is the hardness of approximate solutions to most problems that could have strong impact in the real world?
[1] I did, in passing, find a similar idea discussed somewhere afterwards. If someone has a reference for that, I would be glad to have it.
I'll instead give you something to consider : * The totality of mathematics is not fully defined. As such, there are can be completely new mathematics that destroys previous conceptions especially as it relates to the limits of computability. Theoretical Limits often reflect one's limited scope of perception. Your limits or the limits someone theorized ages ago aren't necessarily my limits if I find a path beyond them.
Information theory and computational complexity theory are theories... Solving fundamental aspects of intelligence will invalidate a host of theories, create new mathematics, new algorithms, and new theories. Continuing to frame something as ground breaking as an understanding/implementation of intelligence in yester-years theoretical limits is flawed.
[1] https://aiimpacts.org/coordinated-human-action-example-super...
In addition, if AI posesses sufficient computing resources, which will certainly become available in the next few decades if not already, it will have inherent strong advantages like serial computation speed and memory size that exceed any human brains. So the real barrier for AGI is software and not hardware.
If AGI software is developed before we have sufficient hardware to run it at the human brain speed, then it will become more capable at the rate of hardware we can put into use, which is likely exponential given how parallelized the human brain appears to be.
The major counterargument I find most convincing regarding outsized impact of exploding intelligence is that many problems are exponentially hard (or harder than that) and thus exponential intelligence can only make linear or sublinear progress on them.
However, linear or even sublinear progress may still lead to quite drastic changes in the world. If an organization can marginally predict stock price movements better than the rest of the world, in a few decades it will accumulate great resources and power. The same is true for many other important domains.