The EA community aren't even unified in minimizing suffering. Most of the formulations I've seen are about maximizing good life, though there are certainly people who care about suffering in the community. (I think it's mostly Tomasik.) I haven't seen any of them try to justify their preferences with evpsych though.
> I'm glad you brought this up, because it raises what I see as the key question, the crux of the matter: "Less Wrong" relative to whom? And for what end?
Once more: Less Wrong relative to a presumed-ordered natural world. To the end of shaping this world to be more in line with one's preferences.
The "natural world" doesn't have beliefs, the world itself cannot be right or wrong, it just is. So the question again is, Less Wrong relative to whom?
> To the end of shaping this world to be more in line with one's preferences.
And what if one is unsure of one's preferences?
What?
Okay, let me try really small steps.
We assume the natural world is ordered.
We assume that the brain holds beliefs, which are compressive/predictive patterns, about the observations of the natural world.
Inasmuch as these are predictive, and inasmuch as we can take multiple actions, then a belief can be qualitatively better or worse in selecting actions that make the world be more good by one's judgment.
Beliefs can fail to be appropriate to select actions on (at least) two metrics: they can be "right or wrong", or they can be "more or less useful."
An example of a wrong belief: if you think it is raining, and so you bring an umbrella, but it is not raining (the belief is wrong), then you have failed to improve your position: you are dry anyway, but now you are also carrying an umbrella, which is annoying.
An example of a useless belief: you can believe that your shirt has more fibers than the average shirt of its kind. This may be true, but even if it is true, there is no decision that you would make where the belief is instrumental in deciding which of the possible actions leads to a world that better matches your preference.
(Unless you have very specific preferences about clothing.)
The goal of becoming Less Wrong is for your brain to contain fewer beliefs that are wrong, and more beliefs that are right; and also (but less importantly) fewer beliefs that are useless, and more beliefs that are useful. The term "Less Wrong" does not refer to a comparison to some other existing human, but rather to the counterfactual outcome of failing to reflect on your thoughts - Less Wrong is a collection of techniques whose goal is, when applied, to make your brain have less wrong beliefs and less useless beliefs, in order to improve your ability to shape the world to your preferences. (Whether they actually achieve this is a hot topic of debate - but that is the standard which they set.)
> And what if one is unsure of one's preferences?
That depends on your preferences for your preferences. :)
In any case, setting your preferences is not the purpose of rationality. (Though as your preferences are part of the ordered natural world, rationality also aims to improve your ability to shape your preferences, this does not tend to be a central consideration.)
But in essence, yes.
(Though of course, again, the ultimate goal is to be better able to achieve preferred outcomes in the world, and if thinking only the word "blue" on repeat produced outcomes perfectly in line with your preferences, then rationality would espouse blue blue blue blue. But at that point, most of the techniques gathered under the label would be useless, because yadda yadda ordered world etc.)
Basically, I am personally of the belief that any "wrong" or "useless" beliefs that one holds are always actually useful, as all belief is instrumental. Ignorance is maintained because it serves a purpose.
I am also of the belief that the only way to determine why one holds a wrong belief is to understand the "emotional" or desire-based reason that underlies or supports it. Hence why I spent a lot of time above discussing emotions. We have seen in recent times the repeated failure of individuals to adjust their beliefs even in the face of overwhelming evidence. Why is this? My stance is that their "wrong" beliefs serve deep emotional needs for them. Most people have stronger desires than the desire to arrive at truth in their evaluation of the world.
The difference in emphasis here is shifting from "the brain as a prediction-making device" to "the human body including the brain is a system that acts to relieve tensions placed upon it", and that prediction-making is one biological strategy for doing so. Of course, better predictions can and will relieve more tension than worse predictions, but my stance is that being able to conceptualize the causes and nature of tension directly will lead to better outcomes, as it permits stronger meta-cognition.
I see the term "preference" doing a lot of work in rationalist thought, and I generally taboo it. The term comes from economics and fails IMO to adequately describe the nature of desire. But if I were to use the term in this instance, I would claim that disentangling one's preferences from internal conflict is the first step one should take prior to tackling the problem of rationality (because we need to know what we even want to get done before we figure out how to do it). This involves interrogating our beliefs about what our preferences are and why, which is something that the rationalist focus on the "world" seems uniquely unequipped for dealing with.
I hope this all makes sense!
(I'm having a lot of fun with this debate too, it's really forced me to firm up my understanding of faith and reason.)
> But if I were to use the term in this instance, I would claim that disentangling one's preferences from internal conflict is the first step one should take prior to tackling the problem of rationality (because we need to know what we even want to get done before we figure out how to do it). This involves interrogating our beliefs about what our preferences are and why, which is something that the rationalist focus on the "world" seems uniquely unequipped for dealing with.
I agree with this take! Though I think it's less that this is something rationality cannot do as that, well, Eliezer when he was writing the Sequences, had an overriding belief that put all his other preferences at risk. This tends to focus one's attention on strategy. :)
The most illuminating post on this, for me, is https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-... . You see this notion of "rationality is a tool to come to conclusions when the conclusions really really matter" all over his writing. I especially like the phrasing in https://www.lesswrong.com/posts/wustx45CPL5rZenuo/no-safe-de... :
> No one begins to truly search for the Way until their parents have failed them, their gods are dead, and their tools have shattered in their hand.
Which implies that to most people, the pursuit of individual rationality, possibly even above mental wellbeing, is counteradaptive.
(Man, does Eliezer have a way with words though.)
(To me, personally, the idea of holding wrong beliefs is like a persistent itch or annoyance. I do not have a driving need, but I also have few attachments. And, of course, I am a nerd, and this is all nerd cocaine. All that sums up to create an ideological space in which I am comfortable.)
This part particularly resonated with me:
> Your trust will not break, until you apply all that you have learned here and from other books, and take it as far as you can go, and find that this too fails you—that you have still been a fool, and no one warned you against it—that all the most important parts were left out of the guidance you received—that some of the most precious ideals you followed, steered you in the wrong direction—
> —and if you still have something to protect, so that you must keep going, and cannot resign and wisely acknowledge the limitations of rationality—
> —then you will be ready to start your journey as a rationalist.
My trust broke years ago, and I found the rationalists, and then I realized, unlike Yudkowsky (I guess?), that I didn't have something to protect. So, my personal goals are in that sense very different.
In my experience with the rationalist community as well, I found that most people I met didn't have a specific personal motivation beside their beliefs surrounding x-risk, and I personally find the set of axioms required to accept rationalist beliefs on x-risk a bit... difficult to swallow.
(Also, AI X-Risk makes a lot more sense if you're already habituated to thinking of the mind in terms of algorithm. 40% programmers: not a coincidence.)
I'm also a programmer and am very habituated in terms of thinking in algorithms, but when I had my "break", I realized that my beliefs were not working on a very deep level, and I needed something a lot stronger/deeper than rationality to push me out of whatever local extremum I was trapped in.
(I think we've run out of disagreements. Good talk?)
(Woe! A deeper rift than even incompatible epistemologies: incompatible chat clients!)
I brought this up because the EA community seems very closely associated with rationalist ideas. Although I'm sure there are other rationalist conceptions of "the good" out there, I've yet to see them clearly articulated.
> ...emotions...
Perhaps the word I should have used is "desire", as emotion springs from desire, and although specific emotional states are easily observable, the underlying desire is what often remains hidden, subject to the other's fantasies. But it's an aside at this point.
> That's rather why it's called "Less Wrong" rather than "Objectively Right".
I'm glad you brought this up, because it raises what I see as the key question, the crux of the matter: "Less Wrong" relative to whom? And for what end?