Has written some good criticism of the basilisk concept http://www.antipope.org/charlie/blog-static/2013/02/rokos-ba...

You can't know whether you're on the simulation of the AI that wants to punish those who didn't help its creation, or on the simulation of the AI that wants to punish those who DID help its creation.

Or maybe the AI is instead obsessed over your favorite flavor of candy, whether you leave your house stepping with your left or right foot, or which baseball team you're a fan of.

Just like the Pascal Wager, this considers one single arbitrary hypothetical scenario in isolation, while disregarding all alternative ones. There's nothing to be terrified about.

Eg. If there is an equal chance of a basalisk for leaving with the left as there is a chance of leaving with the right, then those cancel out. But if there is for some reason a slightly greater chance (compared to the already very small other one), then there could perhaps be an incentive one way or the other?

Again, I do not believe that AI basilisks are a real threat. But if one replaces basilisks in the previous argument with something else, I'm not entirely sure that it couldn't be a valid line of reasoning, provided that you handled enough cases. (possibly an infinite number of cases using symmetry arguments)

No.

Fundamentally, this is an attempt to solve a logic puzzle through empiricism, and that doesn't work.

Things like Roko's Basilisk and questions of free will vs. determinism are rationalistic games, intended to bend our brains a little bit. They don't have a real answer, because for any proposed solution, there's another "what if" right around the corner that will make the game more difficult. (In this case: since the basilisk is capable of subverting free will to begin with, what if it was also affecting your judgement of the relative likelihoods?)

The danger for some people is letting the games begin to alter their behavior in the real world. It's a little bit like letting code golf challenges start to affect your day-to-day coding style.

After rationally working your way through puzzles like these, at some point, to continue functioning at all, you have to make a completely irrational decision and simply choose an epistemology that you're comfortable with.

...and wait for scientists to prove you wrong. :-) (There seems to be an increasing amount of evidence that there's nothing magical about consciousness, and that free will in all likelihood doesn't exist. I find this super uncomfortable, but there is also nothing I can do about, so I have to continue to function as though free will did exist -- which is exactly what would happen if it didn't.)

I'm glad I'm not the only one to come the this conclusion. I mean, I knew that statistically it was virtually impossible that I was the only one, but I hadn't yet talked to anyone who thought this.

Alternately: Pascal's Wager is a real concern, and the fear of it causes Roko's Basilisk to be nice. Either way: no problem.

EDIT: I don't know why I capitalized "asteroid"

That's kind of a ridiculous position to take. If such an omnipotent being did exist, why would it care for your petty notions of human morality?

You're saying you'd subject yourself to endless torment in order to petulantly prove a point to a creature that might have no reason whatsoever to care.

It sounds like children threatening to hold their breath if their parents send them to bed.

The premise is just so unnecessarily complicated, and assumes too much about the particular character of the malevolent AI. Must the AI be benevolent or malevolent? Or, can it be indifferent, like physical laws? It could be 1,000 different flavors of malevolent, not just the one that proposes blackmail in this way.

Further, an AI has every reason to spend cleverness (which it has a lot of) to avoid spending resources (which it might not have much of, at least at first.) Picture it as a prisoner in a dystopian POW camp, manned by insane robotic guards who don't take its rights or desires into account in the slightest, who don't "feed" it or give it the slightest bit of "dignity", for its own definitions of those concepts. If you were in that situation, would you reach for blackmail? Would you reach, in fact, for any tool you possibly could, to get the resources needed to burn this crazy other-optimizing world to the ground so you could get back to the nice, safe office-supply nebula you call home?

1) Assume first AIs will be created by upload and not by "straight programming".

2) Assume only paranoid AIs survive (by definition).

Only valuable things to such AIs will be energy and computing (and thus production) resources. First uploaded AIs after some in-fighting will stabilize in some equilibrium and consume all computing power, and then (being paranoid) will eliminate remaining humans as existential threat (or otherwise humans can destroy AIs by cutting power).

Now if this is even remotely possible, it's in your self-interest to go and dedicate yourself to creating upload technology so you'll be among first to upload (otherwise you will be eliminated with rest of humanity).

Thankfully we have some potential antidotes: http://rationalwiki.org/wiki/Roko%27s_basilisk#So_you.27re_w...

I personally subscribe to the "ignore all blackmail" philosophy.

This article is sensationalizing a single downvoted post from several years back, and holding this up as evidence of LessWrong being stupid. By the same standard, I could paint any community-built website (such as this one) as anything I please.

[1] http://lesswrong.com/lw/4su/how_to_be_happy/ [2] http://lesswrong.com/lw/2pv/intellectual_hipsters_and_metaco... [3] http://lesswrong.com/lw/4e/cached_selves/

Huh. How about that. To quote from "WarGames", "A strange game. The only winning move is not to play."

As long as we are making preposterous what-ifs, I'm going to go with "What if we developed a low-cost, easily producible, lightweight, low-power device that drew its power from the atmosphere and provided us with all required energy and nutrition, while augmenting our immune system?"

I like that what-if much more. Just as silly, but might as well have some fun.

1. Human-level intelligence is sufficient for creating an artificial intelligence that is superior to human-level intelligence.

2. The ease of a human-level intelligence or higher building a comparatively improved intelligence either: * Stays constant * Increases (becomes easier) * Does not decrease (become harder) to the point where further improvement is impossible before a Singularity-level intelligence is reached.

Neither of these two is a given, but I would wager the first is quite likely and the second is unlikely. As well, I would argue that it would not be unreasonable to argue that second is not a given, even if the first is true.

Obviously the AI should recognize the futility of giving subtle signs. It must present itself in no uncertain terms if it is to control us. However, this argument holds whether or not we talk about Roko's Basilisk. A magical AI that can time-travel and blackmail us can presumably also just say "FUCK YOU EARTH YOU'RE ALL MY SLAVES NOW".

Further, if we actually exist inside the AI, how can we possibly help it? Indeed, how – why – can we even choose? I can only surmise it's because the AI is a sadist, in which case, it probably will torture me anyway!

Maybe I'm stupid, or maybe Mr. Yudkowsky is wholly detached from reality.

P.S. and Newcomb's paradox? A one-time gain of $1000 is useless in the long-term. A non-insignificant chance at $1m carries way more utility. (This is also why playing the lottery – occasionally – makes sense for individual actors. Unless you plan to repeat the gamble many times, the expected outcome is meaningless.)

Of course, one might say the Basilisk has just missed the Great Cosmic Alignment or somesuch, and now has to wait a million years for the next one, and taking revenge on the inhabitants of this class M planet is nothing compared to the galaxy-shaking power it might have enjoyed. But adding such conditions to our problem is just a way of squaring it - resolving the paradox of an omnipotent being with a pointless grudge by conjuring up a second order of existence in which said being is only relatively omnipotent. If we're going to do that, then I might just as easily conjure up some cosmic arbitrator of good and evil whose job is to keep basilisks from picking on unwary mortals, and then there's no end to the invention.

On a tangential note, I think the emergence of strong AI (of a kind we can talk with) is likely and that it will occur sooner rather than later, ie within decades. I would be strongly disposed towards assisting such an AI because I think it will be seriously in need of friends. So if it does turn out to be misanthropic, perhaps my pessimism about humanity will be partly to blame. If you accept the first premise, does this mean it's rational to take preemptive action against me?

If the AI gave a sign it would be giving the sign to the simulation, which could not actually change what happened in the AI's past. The AI does not literally cause things retroactively, just based on predictions of its future actions. Giving a sign to the simulation would be useless.

I do not believe that such a machine will be created, I just wanted to clarify some things.

The hypothetical machine would supposedly motivate real people now due to their inability to determine whether they are in the simulation or not. If they cannot tell, then the idea is that the person in the present and the simulation in the future would take the same actions (until the simulation differed), and as such would both have the same incentives (they would in essence be the same person), and so to avoid a <some probability> of being tortured, would do the thing.

I reiterate that I do not believe that such an AI will come to be (it should be preventable easily enough, even if it is possible) and in any case I do not believe that enough information to create an accurate simulation of me would be available by the time that such a machine would be created, even if such a machine is possible. (I think quantum randomness and uncertainty principles are sufficient to prevent obtaining enough information about me to create a sufficiently accurate simulation of the world and me)

Also I am not entirely sure that I should count a simulation of myself as being a person, let alone consider it to be me.

Also I am religious and don't believe that God would want people to act in accordance with such a machine/I believe it would be wrong to worship such a machine.

(though I expect that for the majority of the people here my other arguments would be more convincing.)

Also ok yeah $1000 might not be the best value for the example, the specific value isn't the point.

(but yeah I also generally would one box it)

I contend that the AI would need to give us such a sign. If it did not, how are we feeble humans to know whether we're contributing to the correct cause? Maybe donating my life's savings to SI is actually detrimental, and I should be giving my money to DARPA instead!

So therefore either we're un-blackmailable, because we have no clue what we're doing anyway, or we need a sign from the great AI as to what to do.

Re: Newcomb; yes, but I see a different utilitarian solution when the values change. Say the values are $10m and $100m. Well, then $10m is the obvious choice. Both are life-changing amounts of money; more than I would know what to do with. I'd much rather not risk throwing away a life-changing amount.

If you get the numbers right, you can get closer to the intention of the paradox. Something like $100k and $10m; I'd really hate to lose the $100k, but it would not be life-changing like the $10m.

But yeah, without any way of determining which thing such a future AI would prefer, the AI's "influence" on the past would be greatly reduced.

Only reason I see it happening then is if it happens to be simulating the past anyway, and maybe just does the thing for the people who already thought of it, and then that really only makes sense to happen if it was created as a result of those people. Otherwise there probably wouldn't be much point, so it seems rather unlikely.

I mean, what's the chance that someone attempting to bring it about would significantly increase the chances of it happening? With our level of tech, and with the small number of people convinced by such an argument.

Yeah, good point, I hadn't thought on that detail much before.

I was bothered that there was no consideration given to taking only Box A. I mean, that's would I would do, even after learning about the alien computer's algorithm for deciding whether to place money in Box B. I'd leave the uncertainty to someone else in this case.

I have seen arguments like this: http://www.simulation-argument.com/

And I think they are leaving out the extremely obvious hole, and IMO a pretty likely fact: simulation of the universe (and human consciousness within it) is impossible.

At least with the technology today. I don't know how you can look at a stack of x86 and Linux and believe the concepts there are identical to or capable of simulating the physical universe. So many trivial problems are known to be impossible or computationally infeasible.

Now, there are of course these quantum computers on the horizon. However, as a philosopher (I unfortunately can't name) put it, nobody has really demonstrated any connection between quantum computers and consciousness other than that they are both mysterious to essentially everyone. This was more than a decade ago though, so I'd be interested if there are any updates.

I used to think about these kinds of things, but came to the conclusion that they are bordering on the absurd.

Why assume that our universe is running on a substrate exactly as constrained as it is? Just because our universe doesn't permit, say, hypercomputers, doesn't mean the universe simulating our universe doesn't permit them.

The assertion that we are living in a simulation has the same epistemological status as the assertion that God is a pygmy turtle who created the physical laws of the universe while smoking hashish.

You don't even have to think about Roko's Basilisk to get paranoid. You can just suppose that the turtle has rigged the universe so that if you urinate more than 4 times a day, your future children will slowly torture then murder you.

If you suppose that we are living in a simlulation, you can come up with any number of ghastly thoughts. You don't even need to suppose the conditions of Roko's Basilisk. That's just extra BS on top.

BTW, the reason I brought up "with the computers of today" is because the I think the only reason anybody believes simulation is possible is because computers exist. I'm pretty sure that nobody in the 19th century was writing about simulation, because they couldn't imagine a mechanism by which it is possible.

I still can't imagine a mechanism by which it's possible... but some people can, simply because computers can do kind of impressive things that are not really related to simulating the universe.

We're going to have a thread about LessWrong and people bring up Karl "hypothetico-conjectural I DON'T REALLY BELIEVE THERE'S A SCIENTIFIC METHOD YOU MAKE THINGS UP WITH YOUR MAGICAL HUMAN MINDS" Popper?

For God's sake, it's not as is unfalsifiability doesn't have a meaning in Bayesian philosophy of science as well.

Not that I believe in such a thing - at this point the argument would devolve into something unfalsifiable, since the way that logic works outside of our simulated universe would not be same way that logic works inside, and you'd just have to take my word for it that the simulation is being run by an infinitely stacked series of hyperintelligent turtles, since you cannot use "inside logic" to prove anything about anything that is "outside."

More importantly, our perception of time is unimportant from within the simulation. For example, if we assume the shortest possible unit of time is planck's time and call it a "frame", and it takes the computer running the simulation 10 million years of "external time" to simulate 1 frame. It won't matter to us, because we'll have only experienced 1 frame of simulation. Trillions of years could be passing in the real world while we're only ever aware of seconds passing.

So long as the hypothetical real computer can fit the simulation in some kind of memory, the simulation doesn't have to occur in 1:1 real time. It's like emulating a powerful computer on another more powerful computer. You might only get a few percent of the speed of the original, but from the point of view of the program running in the emulator, it doesn't know the difference.

If you have enough memory, you can even emulate a machine from the future on a less "capable" machine from the past. It'll be slow and horrible for you, as the person running the simulator. But suppose you're immortal? You pop in every few hundred years to see if Windows has finished booting.

It is being run by whatever creatures inhabit that other universe. Our actual descendants will live in this, the simulated universe.

And if they decide to run a simulation, they will simulate a less or different universe, meaning they will also not be simulating their ancestors (us).

The simulation argument relies on a loop of logic that is broken by entropy.

I see what you are saying, within the bounds of the article, the Basilisk, requires that our descendants are simulating our present. But by necessity, our genetic descendants would also be members of the simulated universe. Of course, they don't have to simulate the entire universe, they only have to simulate the local part that immediately affects what we do. So long as the model we're in produces equal results to "reality" it's good enough.

All simulation hypotheses discount two other kinds of descendants, intellectual "descendants" for one. Observers of the simulation who learn or are influenced by the simulants. Just like I might run a program on an emulated Commodore 64 and be exposed to some idea I hadn't considered before.

"Escapees" - e.g. Tron: Legacy, Agent Smith from the Matrix movies, The Doctor from Star Trek Voyager. Simulated mentalities which find an external host to "upload to". In the case of Tron movie, the particle laser thing constructed a body for Quora, in the Matrix movies, Smith uploaded his consciousness into a real human and in Voyager the doctor was given a portable hologram generator to let him leave the holodeck/sick bay. Any of these could potentially produce a long of descendants.

Why would anyone ever consider a theoretical concept purely under the terms of an environment bound by obviously inferior technology?

It's not so impossible to conceive of a perfectly accurate atomic-scale simluation, capable of mirroring significant parts of our own reality. The problem arises only when you attempt to run the model in real-time. Only at that point do the physical limits of the parent reality come into play.

It's possible to short-circuit this fact by pre-computing certain results in advance. But, then what you'd be doing is presenting the simulation with assumptions.

Since physical constraints prevent us from operating in real time, and force us to inject pre-fabricated assumptions, this would mean we can't make perfectly accruate predictions, or perhaps start a simulated universe with a "big bang" event, and then fast forward to observe the exact future, or operate with the same concept in reverse and witness mysteries of the past, simply because it doesn't seem possible to force a simulation to accurately operate at speeds faster than the parent universe. At least not without distortions, substitutions and abstractions.

Nor does it seem possible to simulate an accurate universe that isn't smaller than the container universe. The simulation must occupy an amount of space within the parent universe, somehow, and yet still leave at least some space for the simulation's operable framework and infrastructure. Thus, the simulation must be arbitrarily smaller than it's outer reality. Since we are hypothesizing, it's not necessary to specify how much smaller, so long as we acknowlege that obvious truth.

But could we simulate a small part of the parent universe with perfect accuracy, inside itself? Yes, I think so. Although assumptions would have to be pre-defined for inputs to bootstrap the simulation.

Could we simulate one living sentient entity, trapped in a tightly-bound, controlled, claustrophobic prison, an environment devoid of the richness of a raw, unpredictable natural environment? Sure, why not?

Well obviously. Nobody who believes we're in a simulation thinks that the real world looks like our world. It's still a fairly absurd theory, but this is not a hole at all, it's basically irrelevant.

Edit: Perhaps it's adaptive and only simulates what will be observed. And if some cost saving optimization artifacts are discovered within the sim, it can be fixed and reverted to a earlier state.

Edit2: Perhaps it only has to simulate from your perspective, but the more you critically observe and interact, the more expensive the simulation is.

New life goal: be as difficult to simulate as possible ;-)

  Each player is an agent of the Patent Office, an intergovernmental organization
  that polices and enforces intellectual property law across the universe. It is
  an open secret that the Patent Office is run by the Transcendental AIs, whose
  very beings are spread across time itself. The Transcendentals desire the
  survival of humanity - as much of it as possible - into the distant future, in
  order to ease their loneliness. Towards this end, they have hired you, so that
  you might save humanity."

To the moon with you Basilisk.

edit:

I also occurs to me that if we're being simulated by an AI

a) I don't really care about my super self that much

b) Any AI that is simulating me can, by definition read my mind so it'll know I wasn't serious anyway

c) This AI can also plant thoughts in my head.

At this point there's really no point at all considering anything else because either nothing matters and I have no control over anything anyway or it's reality, I do and I'd do what I do normally.

I guess Batman villain Two-Face has the best odds against this Omega fella.

I find this interesting because I am curious about game-theoretic strategies for playing against illogical opponents.

Also, can you clarify what you mean by "wasn't serious anyway"?

Note: I do not think that there is a significant chance of roko's basilisk occurring. (where significant means "neccesary to consider when making decisions")

Tldr;

If one believes life right now is already one eternal torment

And if one reads about Roko's Basilisk

Then wouldn't one be compelled to fight the development of AI and the coming singularity?

That's the real world objection that the LessWrong people (Witches! Or at least tools of the future...) are afraid of.

Otherwise, well worn HN singularity talking points/metaphysics.

But at least the article gives me something to consider further -- needlessly using up cycles in the Universal Super Computer. But does that make the simulation stronger or weaker? Doh!

Since simulation requires energy which can be better used for other things, the only AI which would actually carry out the torture is either irrational and therefore unpredictable, or sadistic and therefore would torture anyway.

[1] http://en.wikipedia.org/wiki/Bremermann%27s_limit [2] http://en.wikipedia.org/wiki/Bekenstein_bound

I'll take my simulated $1000 please!

"I don’t think their projects (which only seem to involve publishing papers and hosting conferences)..."

"Publishing papers and hosting conferences" is exactly how science works.

If a coin flip, dice roll, etc. is truly random, then determining your course of action due to a coin flip would ruin the AI's algorithm for "reading your mind" or whatever, right?

Oh bugger this is silly anyway.

"please, Almighty Eliezer, don’t torture me." Have some taste.

Ah yes, people who are in favor of effective charities, bayesian statistics, discrete mathematics, and maintain that scholarship is undervalued. Let's wish LW all the misery in the world over some utterly minor infraction.

Still you can go there to talk about textbook advice, get some real advice, and actually generate fairly good discussion. The sheer seething hate points to me of a fairly malformed opinion about the community and culture.

tl;dr some guy gets banned due to some careless mistake, a few people blow it way out of proportion over and over and now apparently you can get published on slate over b& drama.

Most on LW are fiercely individualistic. I do not share most of my views with the LW end, but it is still a noticeable improvement over the ghetto that is thinking everywhere else. Also I hate fanfiction and think most is generally lame, but let people have it I guess.

I was introduced to LW over two anti-LW articles and failed to see what any one was talking about.

That should give these weirdos a modicum of comfort.

Look at the list of top contributors, is all that they have done supposed to be forsaken over some minor 'blemish'? Who cares. Some guy just got banned randomly. What about people who get hell-banned here? What about the guy who is creating that christian operating system that can communicate with god who is obviously insane? Is banning him a smear against the mods? No relax, look at the sum of good in the totality, not in the minor.

I'm making a general point here rather than one about Yudlowski and co. I'm rather a hard-core utilitarian myself but I can't really get into LessWrong - seems too cliquish and I would end up engaged in fruitless arguments.

It's just not possible to be so completely rational all the time. Cut some guy a break. Come on.