In order to say things like "my brain" one is necessarily creating a distinction between themselves as thinking subject and a "lower self" which consists of a biological substrate. Of course, this is common practice in folk epistemology, but in order to achieve this, one needs the support of an "objective" frame of reference, which can only be achieved through a theological justification (or in this case cosmological, through reference to evolutionary processes).

This was an issue that Kant grappled with, in that the mind cannot distance itself from itself. When we "understand the game theoretic strategy that our brain is executing", we are still implicated in other "game theoretic strategies", we've just made certain ones manifest and chosen to ignore others.

That said, I still support the end-goal of "bringing emotions into harmony with our conscious beliefs", but often that requires turning in the opposite direction: asking questions to the emotions themselves rather than to the world, which requires an entirely different conceptual toolkit (e.g. psychoanalysis, Buddhist thought, etc).

I view evolutionary psychology as in the category of psychological divination techniques, in which a set of predetermined systematic constraints allow you to locate interesting patterns in your day to day life through a process of reorientation. In all sincerity, I suggest astrology as an alternative technique that avoids any normative connotations of "fitness" etc.

> In order to say things like "my brain" one is necessarily creating a distinction between themselves as thinking subject and a "lower self" which consists of a biological substrate.

Come on, that's a stretch. I think it's clear from context that this is referring to unconscious vs conscious awareness, in the like one case where I failed to specifically highlight this distinction (because I wanted to head off just this claim). Both of course happen in the same slab of meat.

> This was an issue that Kant grappled with, in that the mind cannot distance itself from itself. When we "understand the game theoretic strategy that our brain is executing", we are still implicated in other "game theoretic strategies", we've just made certain ones manifest and chosen to ignore others.

Sure, but consider: [1] [2]. Of course conscious deliberation is just another strategy, but it's a strategy that benefits from clarity. There is no good reason why reason and deliberation should be unable to close over itself, let alone as "separate" a component as emotional state.

Reason is a strategy that rose to some prominence in human evolutionary history because it is extraordinarily effective at making sense of the natural world. (Note: There's some claims that reason rose to prominence because it made humans good at making up sensible claims to deceive others - but this sort of evolutionary history would still produce a system that's good at inference, if only as a side effect. Compare GANs.)

> that requires turning in the opposite direction: asking questions to the emotions themselves rather than to the world, which requires an entirely different conceptual toolkit

Now who's doing dualism? The emotions are part of and arose from within the world. Without looking at the historical context in which they evolved, you may not be able to even understand what they're trying to tell you.

And sure, a lot of evpsych is just-so trash. But especially around the game theory of emotions, I think the conclusions hold up.

[1] https://www.lesswrong.com/posts/46qnWRSR7L2eyNbMA/the-lens-t...

[2] https://www.lesswrong.com/posts/XPErvb8m9FapXCjhA/adaptation...

> There is no good reason why reason and deliberation should be unable to close over itself, let alone as "separate" a component as emotional state.

It can be done, but most rationalists never achieve this, in that it requires understanding the cause of emotion as it relates to one's own subjectivity. This often involves admitting hard truths about oneself, and is the process that psychoanalysis is intended to support. Instead they "rationalize", and come up with explanations like that one that follows, which don't understand the emotion so much as "explain it away" through what amounts to, as I said earlier, a metaphysical argument about the causal origins of the species.

> Reason is a strategy that rose to some prominence in human evolutionary history because it is extraordinarily effective at making sense of the natural world.

This sort of causal argument is why I tend to cringe at rationalist theory, because not only is it not necessarily true, there's no way to even tell if it were true.

> The emotions are part of and arose from within the world. Without looking at the historical context in which they evolved, you may not be able to even understand what they're trying to tell you.

Ironically I fully agree with this statement, except what I would mean by "historical context" is "the events in your own specific life that influenced your current sentiments", rather than history viewed as a totality. Again, a reference to the totality and causal origin of life; do you see why I describe it as theological or metaphysical? This is very similar in structure to Christian theories on the topic of e.g. original sin.

> This sort of causal argument is why I tend to cringe at rationalist theory, because not only is it not necessarily true, there's no way to even tell if it were true.

I mean, to be fair I'm leaning myself out the window a lot with that claim. But it seems plausible, and more importantly, the argument is resilient to why reason arose. I'm not saying this in a sense of "this is definitely why we have reason, and that's why I am right"; I'm more saying "even in this case where we don't necessarily have reason because it's useful," (note the alternate theory I list) "that is still no reason (heh) to expect it to mislead us." And I think we can confidently say that reason is useful today, at least, and submit as evidence approximately the last 10000 years of planetary history. I think it's not a big stretch to claim that the reason that reason is useful today may be the reason that it evolved to begin with.

> Ironically I fully agree with this statement, except what I would mean by "historical context" is "the events in your own specific life that influenced your current sentiments", rather than history viewed as a totality.

And again, I don't see why it has to be one or the other. Of course, even and especially in "events in your own life" it is ironically very easy to make up pat just-so stories to justify any reaction or sentiment. In comparison I honestly think evpsych is on a firmer footing, particularly given I don't think psychotherapy needs particularly to be correct about the causes it posits in order to function as therapy. Whereas if you are mistaken about the evolutionary cause of an emotion, you will completely misconstrue it and probably behave badly.

> It can be done, but most rationalists never achieve this, in that it requires understanding the cause of emotion as it relates to one's own subjectivity. This often involves admitting hard truths about oneself

For instance, I am very leery of the notion of "hard truths", because it imbues an argument with value based on the emotional work performed in reasoning it out. This is the exact sort of thing rationality says should not be expected to work. - Something does not become more or less true by being hard to admit; an untruth can be as hard to admit as a truth. You need to have a cognitive mechanism that is in principle capable of producing truths to begin with, in order to even arrive at a hard truth that you can emotionally reject. Without that, you're liable to just admit hard made-up claims as a form of catharsis.

Is something being hard to admit evidence that it is true? Yes, but weakly - if it weren't convincing enough to outweigh being hard to accept, you probably wouldn't even entertain it. (How to be convinced by things that are true but hard to accept is actually a core tenet of rationality - see How To Actually Change Your Mind linked above.) But you still have to do the work to arrive at a true belief about yourself to begin with.

> "even in this case where we don't necessarily have reason because it's useful,... that is still no reason (heh) to expect it to mislead us."

There are plenty of cases of reason misleading, going wrong, perhaps as many or more as reason going right. But of course, judgment of right and wrong depends on your criteria, and reason will never help you achieve that which is prior to reason: faith, in the most basic form.

But, in rationalist theory, by positing itself as the means and also the end (we use reason to get closer to reason; we have faith in reason), the term forms the sort of self-sustaining loop that Zizek (or really Lacan) calls a "master signifier", and is also the essence of all theology. All other considerations are shoved under the bus, or else dictated by a sort of bland evolutionary ethical theorizing, like "minimizing suffering", without really interrogating what that even means (if they looked into older philosophy, they would see that thought itself could be considered a form of suffering). It is not necessarily bad to hold to a master signifier as such, as all faith exists in this form, but the rationalists seem especially unaware or unable to interrogate their beliefs to this level.

> I am very leery of the notion of "hard truths", because it imbues an argument with value based on the emotional work performed in reasoning it out.

This is a misunderstanding of what I meant to express, and I apologize for using that term. In my experience (and in the psychoanalytic literature), "hard truths" are "hard" because they involve noticing objective aspects of ourselves that we were previously blind to. The recognition often comes in a flash, with a sense of relief, rather than through some sort of laborious process (of course, the preparation for that flash might well be a laborious process of re-learning how to look!). The "hard truth" is being able to admit something like "actually, I'm not just out to get the truth. I want social recognition after all."

> Something does not become more or less true by being hard to admit; an untruth can be as hard to admit as a truth

Such a focus on truth in relation to psychological affairs feels, to me, wrong-headed, in that all truth is contingent, grounded in some act of judgment. Don't read this the wrong way: contingent != relative. It just means that we need to address the question of "how do I know that true things are true?".

The cultural impact of the enlightenment, especially of Kant, was to collapse "truth" into an intersubjective agreement based on sensory observation ("science"), but how can you achieve intersubjective agreement in affairs which are only observable to a single subject, such as "my emotions" (i.e. "how is psychology possible?")? This is where psychoanalysis had its real epistemological innovation: free association as a technique is a way of generating shared knowledge about what was formerly private. Christian "confession" functioned similarly. Modern behavioral psychology is several steps behind. But with or without such a technique, one can never really achieve the absolute (gnostic) truth about oneself, at least not without a strong frame that answers the question "under what circumstances is a given statement about myself true?"

It seems better to ask the question of "what do I want that I'm having difficulty achieving? And what factual observations about myself can I make, which demonstrate patterns that prevent me from achieving this?" Notice how this moves the direction of thought (of reason, even!) away from "finding the truth" and toward a more environmentally-oriented mode of "understanding", closer to what Plato meant by that term, with its connections to "wisdom". The "hard truth" here as I mean it, is noticing sites of competing desires, as Spinoza and Freud do in their discussions of "ambivalence", and then being willing to take a stance or a risk that attempts to resolve the tensions.

With that in mind...

> I don't think psychotherapy needs particularly to be correct about the causes it posits in order to function as therapy. Whereas if you are mistaken about the evolutionary cause of an emotion, you will completely misconstrue it and probably behave badly.

This is a feature, not a bug, because as I said earlier, the goal isn't to discover some absolute truth about yourself, but to solve problems. Evopsych explanations claim to provide truth, but actually serve to solve problems, through some intellectual sleight-of-hand.

Similarly, the use of reason in thought serves to solve problems, meet individual needs, even if framed as "uncovering the truth". This + my deep involvement in the rationalist world for several years is why I feel comfortable making statements like this one from a few days ago: https://news.ycombinator.com/item?id=28198926

> There are plenty of cases of reason misleading, going wrong, perhaps as many or more as reason going right.

Right, but note what you are saying: reason going wrong.

I don't think I even need to add anything there. Your phrasing already reveals that there is "a rightness ... by which it may be judged" in reason that there is not in faith.

> But, in rationalist theory, by positing itself as the means and also the end (we use reason to get closer to reason; we have faith in reason), the term forms the sort of self-sustaining loop that Zizek (or really Lacan) calls a "master signifier", and is also the essence of all theology.

I agree that reason is ultimately circular. I even agree that religion can also be circular. (I disagree strongly that all religion is circular, or that religion is inherently circular, or even that religion inherently hinges on faith.) Nonetheless, I think that where religion is circular it is so in a different way from reason. When we reject the circularity of religion, we do so not because it is inherently wrong to posit a God that justifies everything, but because it is not useful. The very self-supporting structure of the faith, the retreat to an insulated core element, prevents its adherent from doing any cognitive or predictive work with it. Rather, I would argue the circularity of religion is a consequence of reason, a retreat into self-justifying tautology prompted by the failure of religion to compete with reason on its own grounds.

In other words, reason closes on itself in a way that shapes itself to a particular inherent standard ("that seems to be associated in some way with success in interacting with the natural world", my reason throws in, giving yet another reason why it is judged valuable under itself, argumentatively achieving nothing yet undeterred.) However, religious faith does not even have any such constraint.

> bland evolutionary ethical theorizing, like "minimizing suffering"

I don't know why you are talking about suffering. I was not talking about suffering. I have no idea where you are going with that.

Evolved animals don't seek to minimize suffering. They don't even seek to maximize propagation; they're just the result of a process of relentless selection for things that propagate.

> "hard truths" are "hard" because they involve noticing objective aspects of ourselves

I'm sorry, you were previously accusing rationality of putting too much faith in reason - and now we're supposed to just notice objective aspects of ourselves? With what absolute source of truth, exactly, are we supposed to do this?

> The recognition often comes in a flash, with a sense of relief, rather than through some sort of laborious process

Now, I myself am as much of an addict to this sort of sudden endorphin-supported flash of insight as the next nerd. But - though it is an indicator of a simple theory that supports the facts observed, that does not in itself suffice to establish truth, let alone "objective truth"!

I have a new theory: lots of rationalists are programmers because programmers are used to having the sudden flash of insight be wrong.

> "how do I know that true things are true?".

I mean - rationality in particular does not engage in any attempt to establish truth, so I feel this charge somewhat misses its mark. Rather, rationality seeks to improve the quality of one's beliefs by engaging in processes that interact with the presumption of an ordered natural world in order to come to beliefs that are, given that assumption, more likely to be true.

That's rather why it's called "Less Wrong" rather than "Objectively Right".

> but how can you achieve intersubjective agreement in affairs which are only observable to a single subject, such as "my emotions"

I have a rejoinder to this called "you are not a unique and beautiful snowflake" and also emotions are totally observable to others, like, it's usually not even hard. Doesn't even take blood tests or brain autopsy. When some dude beats my face in with a bottle cause I fucked his wife, I do not have a particular hard time parsing his emotional state - or the evolutionary reasons behind it! Emotions are interesting because they have effects, on yourself (the brain is part of the natural world!) but also on others through you. Honestly, the idea that emotions are primarily individual is just so weird to me. They seem one of the most shared aspects of humanity there is.

> It seems better to ask the question of "what do I want that I'm having difficulty achieving? And what factual observations about myself can I make, which demonstrate patterns that prevent me from achieving this?"

From what you're saying, I take it you have observed rationalists failing at this. But I mean, going just by the text and the people I know, this just seems like solid rational thinking. I don't see any contradiction here.

> I don't know why you are talking about suffering. I was not talking about suffering. I have no idea where you are going with that.

I brought this up because the EA community seems very closely associated with rationalist ideas. Although I'm sure there are other rationalist conceptions of "the good" out there, I've yet to see them clearly articulated.

> ...emotions...

Perhaps the word I should have used is "desire", as emotion springs from desire, and although specific emotional states are easily observable, the underlying desire is what often remains hidden, subject to the other's fantasies. But it's an aside at this point.

> That's rather why it's called "Less Wrong" rather than "Objectively Right".

I'm glad you brought this up, because it raises what I see as the key question, the crux of the matter: "Less Wrong" relative to whom? And for what end?

> I brought this up because the EA community seems very closely associated with rationalist ideas. Although I'm sure there are other rationalist conceptions of "the good" out there, I've yet to see them clearly articulated.

The EA community aren't even unified in minimizing suffering. Most of the formulations I've seen are about maximizing good life, though there are certainly people who care about suffering in the community. (I think it's mostly Tomasik.) I haven't seen any of them try to justify their preferences with evpsych though.

> I'm glad you brought this up, because it raises what I see as the key question, the crux of the matter: "Less Wrong" relative to whom? And for what end?

Once more: Less Wrong relative to a presumed-ordered natural world. To the end of shaping this world to be more in line with one's preferences.

> Less Wrong relative to a presumed-ordered natural world

The "natural world" doesn't have beliefs, the world itself cannot be right or wrong, it just is. So the question again is, Less Wrong relative to whom?

> To the end of shaping this world to be more in line with one's preferences.

And what if one is unsure of one's preferences?

> The "natural world" doesn't have beliefs, the world itself cannot be right or wrong, it just is. So the question again is, Less Wrong relative to whom?

What?

Okay, let me try really small steps.

We assume the natural world is ordered.

We assume that the brain holds beliefs, which are compressive/predictive patterns, about the observations of the natural world.

Inasmuch as these are predictive, and inasmuch as we can take multiple actions, then a belief can be qualitatively better or worse in selecting actions that make the world be more good by one's judgment.

Beliefs can fail to be appropriate to select actions on (at least) two metrics: they can be "right or wrong", or they can be "more or less useful."

An example of a wrong belief: if you think it is raining, and so you bring an umbrella, but it is not raining (the belief is wrong), then you have failed to improve your position: you are dry anyway, but now you are also carrying an umbrella, which is annoying.

An example of a useless belief: you can believe that your shirt has more fibers than the average shirt of its kind. This may be true, but even if it is true, there is no decision that you would make where the belief is instrumental in deciding which of the possible actions leads to a world that better matches your preference.

(Unless you have very specific preferences about clothing.)

The goal of becoming Less Wrong is for your brain to contain fewer beliefs that are wrong, and more beliefs that are right; and also (but less importantly) fewer beliefs that are useless, and more beliefs that are useful. The term "Less Wrong" does not refer to a comparison to some other existing human, but rather to the counterfactual outcome of failing to reflect on your thoughts - Less Wrong is a collection of techniques whose goal is, when applied, to make your brain have less wrong beliefs and less useless beliefs, in order to improve your ability to shape the world to your preferences. (Whether they actually achieve this is a hot topic of debate - but that is the standard which they set.)

> And what if one is unsure of one's preferences?

That depends on your preferences for your preferences. :)

In any case, setting your preferences is not the purpose of rationality. (Though as your preferences are part of the ordered natural world, rationality also aims to improve your ability to shape your preferences, this does not tend to be a central consideration.)

So, to be perfectly clear, the main reason for studying rationality as you explain it, is the belief that many individuals are not skilled at determining whether their beliefs about the observable world are right or wrong (as in your umbrella example), and that this is something they would benefit from working on?

Well, most people can generally recognize when a belief is wrong if they are directly confronted with the fact. I think it's a bit more meta - such as creating modes of thought that systematically (again, holding in mind the assumption that nature is regular) produce right and useful thoughts, and dismantling modes of thought that systematically produce wrong or useless thoughts.

But in essence, yes.

(Though of course, again, the ultimate goal is to be better able to achieve preferred outcomes in the world, and if thinking only the word "blue" on repeat produced outcomes perfectly in line with your preferences, then rationality would espouse blue blue blue blue. But at that point, most of the techniques gathered under the label would be useless, because yadda yadda ordered world etc.)

Thanks for sticking with me as I clarified, I see the crux of our disagreement now.

Basically, I am personally of the belief that any "wrong" or "useless" beliefs that one holds are always actually useful, as all belief is instrumental. Ignorance is maintained because it serves a purpose.

I am also of the belief that the only way to determine why one holds a wrong belief is to understand the "emotional" or desire-based reason that underlies or supports it. Hence why I spent a lot of time above discussing emotions. We have seen in recent times the repeated failure of individuals to adjust their beliefs even in the face of overwhelming evidence. Why is this? My stance is that their "wrong" beliefs serve deep emotional needs for them. Most people have stronger desires than the desire to arrive at truth in their evaluation of the world.

The difference in emphasis here is shifting from "the brain as a prediction-making device" to "the human body including the brain is a system that acts to relieve tensions placed upon it", and that prediction-making is one biological strategy for doing so. Of course, better predictions can and will relieve more tension than worse predictions, but my stance is that being able to conceptualize the causes and nature of tension directly will lead to better outcomes, as it permits stronger meta-cognition.

I see the term "preference" doing a lot of work in rationalist thought, and I generally taboo it. The term comes from economics and fails IMO to adequately describe the nature of desire. But if I were to use the term in this instance, I would claim that disentangling one's preferences from internal conflict is the first step one should take prior to tackling the problem of rationality (because we need to know what we even want to get done before we figure out how to do it). This involves interrogating our beliefs about what our preferences are and why, which is something that the rationalist focus on the "world" seems uniquely unequipped for dealing with.

I hope this all makes sense!

Yeah that makes sense. I really think where this is running apart is that... like, a lot of the people driving the rationalist project are in environments where holding strategically correct beliefs isn't just useful but vital. Eliezer famously believes on a gut level that he once held a wrong belief and that if he had acted on that wrong belief, the world would have ended. I don't know how that relieves a tension, but I can see how it would make you really really wary of cognitive mistakes.

(I'm having a lot of fun with this debate too, it's really forced me to firm up my understanding of faith and reason.)

> But if I were to use the term in this instance, I would claim that disentangling one's preferences from internal conflict is the first step one should take prior to tackling the problem of rationality (because we need to know what we even want to get done before we figure out how to do it). This involves interrogating our beliefs about what our preferences are and why, which is something that the rationalist focus on the "world" seems uniquely unequipped for dealing with.

I agree with this take! Though I think it's less that this is something rationality cannot do as that, well, Eliezer when he was writing the Sequences, had an overriding belief that put all his other preferences at risk. This tends to focus one's attention on strategy. :)

The most illuminating post on this, for me, is https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-... . You see this notion of "rationality is a tool to come to conclusions when the conclusions really really matter" all over his writing. I especially like the phrasing in https://www.lesswrong.com/posts/wustx45CPL5rZenuo/no-safe-de... :

> No one begins to truly search for the Way until their parents have failed them, their gods are dead, and their tools have shattered in their hand.

Which implies that to most people, the pursuit of individual rationality, possibly even above mental wellbeing, is counteradaptive.

(Man, does Eliezer have a way with words though.)

(To me, personally, the idea of holding wrong beliefs is like a persistent itch or annoyance. I do not have a driving need, but I also have few attachments. And, of course, I am a nerd, and this is all nerd cocaine. All that sums up to create an ideological space in which I am comfortable.)

Thanks for sharing those two posts. These might be instrumental in helping a (more rationalist-leaning) friend and I move forward in a years-long debate about ethics, lmfao

This part particularly resonated with me:

> Your trust will not break, until you apply all that you have learned here and from other books, and take it as far as you can go, and find that this too fails you—that you have still been a fool, and no one warned you against it—that all the most important parts were left out of the guidance you received—that some of the most precious ideals you followed, steered you in the wrong direction—

> —and if you still have something to protect, so that you must keep going, and cannot resign and wisely acknowledge the limitations of rationality—

> —then you will be ready to start your journey as a rationalist.

My trust broke years ago, and I found the rationalists, and then I realized, unlike Yudkowsky (I guess?), that I didn't have something to protect. So, my personal goals are in that sense very different.

In my experience with the rationalist community as well, I found that most people I met didn't have a specific personal motivation beside their beliefs surrounding x-risk, and I personally find the set of axioms required to accept rationalist beliefs on x-risk a bit... difficult to swallow.

Yeah that all makes sense. I called myself a "rationality consumer" in a LW post a few years back, and I think that roughly still holds. Though I will say that being part of a community where wrong beliefs are corrigible through argumentative persuasion is an immense social relief for me.

(Also, AI X-Risk makes a lot more sense if you're already habituated to thinking of the mind in terms of algorithm. 40% programmers: not a coincidence.)

This also makes sense. I did enjoy all the arguments I got into on the SSC discord. It was formative for me even though I found myself often frustrated when the arguments reached an impasse, often in relation to the desires of the interlocutors.

I'm also a programmer and am very habituated in terms of thinking in algorithms, but when I had my "break", I realized that my beliefs were not working on a very deep level, and I needed something a lot stronger/deeper than rationality to push me out of whatever local extremum I was trapped in.

Hey, if it works for you it's all to the good. :)

(I think we've run out of disagreements. Good talk?)

good talk! check my about page for a discord if you wanna chat again

Thanks but note, I almost entirely hang out on irc. Don't wanna link here but I'm sure you can guess the channel :P

(Woe! A deeper rift than even incompatible epistemologies: incompatible chat clients!)

ahaha classic... I haven't used IRC since my college days, unfortunately I don't think I can support another chat client tab :P (the software i'm working with takes up way too much memory smdh)