Humorism was a field of academic study for two millennia before germ theory demonstrated how laughable the entire field was. It’s always important to remember that doing research in a field doesn’t necessarily mean that the field itself is on a solid foundation that maps to reality. Psychology and psychiatry really do struggle to demonstrate statistically meaningful results behind most therapies (CBT is perhaps a notable exception, although even there results can be mixed). Analyses like the big 5 are squarely in the “let me read your tea leaves” camp of “science” IMO. They basically play a trick of “sure, the other personality theories were bogus but this time we did stumble upon a good way of characterizing it” while taking at face value that the entire endeavor as valid. Neuroscience has a bit more behind it because it’s actually rooted in quantifiable measurements somewhat and even there it’s difficult. It’s like trying to reverse-engineer a modern CPU from X-ray lithography. If you know the theory underpinning everything, you can start to make educated guesses. Without that you’re not going to get anywhere close to what’s going on let alone root cause if something is broken.

A major problem with humorism was that it did not have any empirical evidence to support it.

Modern personality theory DOES. Personality theory uses a variety of techniques to show that a construct is internally consistent and related to real-world indicators. If independent people can view a video or a person and they reliably rate that entity similarly (inter-rater reliability), we can say with some certainty that there is something there. Relating a construct like this to a variety of real-world outcomes allows us to ascertain its relationship to the world. This evidence is messy and not nearly as solid as what's found in the harder sciences, but your dismissal of the modern statistics of questionnaire-based measurement does a very well-researched field of inquiry a huge disservice.

My analogy to humorism was not to imply that there’s no empirical evidence at all here. Just that the field of psychology has continually struggled to actually develop testable hypotheses that lead to scientific theories.

There’s a giant chasm between “I have a signal” and “here’s what that signal means”. You can’t start out with a data set, pick out some signals and then say what they mean. That’s scientific theory 101. You build a hypothetical model and then design experiments to test those theories. It’s not me doing the field a disservice. The practitioners have consistently and repeatedly done bad science and statistics. I’ll be more supportive once the field actually starts producing falsifiable theories that are supported by evidence and the practitioners are better at calling out their peers.

You seem to be conflating psychology the science vs psychotherapy the practice. The latter approximates engineering and producing real life results takes precedence over proving things from first principles. Just like with medicine, you don't categorically reject medications even though it can't always explain the mechanisms of action fully (yet). Big tech doesn't hold off on making billion dollar business decisions based on the same techniques either. Science has its place but it is not always above real life pragmatism.

It’s remarkable how similar this is to the argument that an acupuncturist will make is. The modern pharmaceutical industry in many instances has the same issues by the way. Lots of drugs that have questionable efficacy at best.

Yes, my dad, a pharmacist, told me a drug just have to be a little bit more efficient than a placebo to pass certification, and he sells homeopathy all day as well (while calling it a scam), because it does relieve people of their pain...

Acupuncture, which is having someone dancing around you while touching you, is like everything: find something cheaper and more efficient to solve the same problems and we wont see it anymore.

> Big tech doesn't hold off on making billion dollar business decisions based on the same techniques either.

I've seen how large companies make ≈$X0 million decisions and probably $X00 million decisions and I suspect $ billion decisions aren't treated fundamentally differently. I wouldn't use that as evidence for how to make decisions well.

My current theory is that companies are successful not because they consistently make specific tactical or even strategic decisions well but because (through design or evolution) they are resilient systems that can weather surprisingly poor decision making at all levels.

Turns out that the big 5 have predictive power. Hard to call it "tea leaves" if it has broad correlates with human behavior that are replicable across cultures and predict forward-looking behavior.

That's the argument usually presented for intelligence testing.

So?

My goodness, is intelligence testing now so taboo that this constitutes an argument?

Nope, just pointing out the commonality.

> Neuroscience has a bit more behind it because it’s actually rooted in quantifiable measurements somewhat and even there it’s difficult.

The problem still remains of how the physical measurements (even assuming we could measure them perfectly, which we definitely can't) relate to the internal experience.

Any real theory of psychology/neuroscience needs to be able to explain that, and currently I'm not aware of any that do.

> Psychology and psychiatry really do struggle to demonstrate statistically meaningful results behind most therapies (CBT is perhaps a notable exception, although even there results can be mixed)

So it's incredibly difficult to do good studies of therapy, because there is no simple placebo. All of the research I've seen suggests that there's huge amount of provider-level variance, which swamps the theoretically meaningful defences. I would suggest that this is because we don't really have a good understanding of how to interrogate our own (or others) psyches, which is definitely not something that neuroscience will help with (given the current state of the field, at least).

Sounds like we're in agreement. What I hear you saying is that the current situation is likely worse than my pessimistic take on it, right? :).

My only disagreement with you was that neuroscience will solve the problem, which it almost certainly won't, at least in my lifetime (hopefully I'm wrong though).

To be fair, Costa and McCrae did a lot of great work on the Big 5, but they really didn't like it when SEM/CFA didn't really support their theory.

In fact, they liked it so little that they invented Procrustean rotation so that they could make the factors come out the "correct" way.

Once I discovered that, I paid a lot less attention to the Big 5.

Also, I haven't checked the source papers, but that all blue stuff could come from a general tendency to agree with the questions/put a positive spin on that. I do know that not all personality measures are balanced on positive vs negative questions, to remove this bias (life satisfaction methods are almost certainly not balanced in this way).

But isn't that just tautology with an intermediate step?

You might have Personality Descriptive Adjective X that is found to cluster into what we call a Big Five trait.

But Adjective X has its own definition, and it's likely that the tautology (at least insofar as these life satisfaction correlations are concerned) happens at that level.

Tautology implies that the definition we give is redundant, but none of these things were defined in this way. This is because none of the research relies on this kind of dictionary definition.

Instead, they literally ask people hundreds of questions of the type:

"To what extent would you consider yourself to be [adjective]?" (0 - 'not at all' to 5 - 'complete')

Then they factor analyze the results to see which adjectives tend to be highly related to each other. Factor analysis builds on correlation to find shared variation among large groups of indicator variables.

So it is related to how people think of the adjectives and what they mean, but it is not the case that we are just lumping definitions together or "saying the same thing in different words" in the way of a tautology. Instead, what we're saying is that "Most of these personality adjectives mostly mean the same thing. Here's the thing they all mostly mean. If humans in all languages have tons of words that mostly mean this common thing, maybe this common thing is significant and worthy of study."

Regarding life satisfaction, the sense that one finds their life satisfactory and "would change nothing" (one of the items in the Satisfaction with Life Scale that was most likely used to measure it) doesn't show up in those other traits in any major way. In fact, to the extent that a description of the self is related to a world-view, belief or other external factor, it will never be a tautology, because a self-view and other-view or world-view will never share a common definition or tautology by virtue of the self-other distinction.

> Then they factor analyze the results to see which adjectives tend to be highly related to each other. Factor analysis builds on correlation to find shared variation among large groups of indicator variables.

The big problem with these results (and god knows I've analysed enough of them) is that the extent that they map to other characteristics measured without surveys is pretty much unknown.

If you want to create a new measure of extraversion, then you use the older methods (i.e. other surveys), and this only really tells you that the new measure correlates with the old.

I vaguely recall Eysenck doing physiological tests for extraversion/introversion which would avoid this problem, but I'm not aware of any more recent research in this area (mind you, I've been out of the field for a decade now).

I suppose they asked 1000s of different questions, each representing a different personality characteristic. After factor analysis, some arbitrary number of factors were found to account for most of the variance. Why did they settle for 5 vs 8 vs 12?

There are a number of metrics that are commonly used to settle on a number of factors to extract. There's some judgment in it to be sure, but you are looking at data-driven phenomena as well like percentage of variance explained, a precipitous drop in new added value, and etc.

Essentially you come to a point where extracting another value isn't explaining much more variance or producing a factor that's meaningful (e.g., two items of conscientiousness with negative wording are the entire new "factor" and only take variance explained from 80% to 82%).

https://www.theanalysisfactor.com/factor-analysis-how-many-f...

Like pretty much every unsupervised learning problem, there is no data which can tell you what N should be. If there was, you could trivially use that measure to convert it to a supervised learning problem.

More generally, all of those metrics are pretty fluffy (the scree plot is probably the best, as it doesn't give you a p-value).

In order to actually do this right, you need multiple studies and CFA, but the standard in the field is to refit your exploratory model as a CFA on the original dataset, which is almost certainly causing overfitting in most cases.

Other models actually did settle for higher numbers of factors, but higher numbers were usually found to have categories that could be reasonably collapsed into one of the Big 5.

Fwiw, my understanding is that most psychologists believe that with further study, these categories will map to actual functioning of the brain and nerve systems. For example, "Neuroticism" will somehow map to the excitability of certain nerve systems and "Agreeableness" will somehow map to functioning of our mirror and mentalizing systems. "Conscientiousness" will map to inhibiting functions, "Openness" will map to approach/flee type emotional functions, and "Extraversion" will map to systems like dopamine release on securing rewards for behavior.

That's a belief that's a long way from being testable though.

So how do we objectively measure all 5 dimensions? I.e. would I roughly assess myself similar to how others (on average) would assess me?

There is a bunch of item pools that has been developed for this: https://en.wikipedia.org/wiki/Big_Five_personality_traits#Me...

Given that the definitions of words and phrases are possibly subjective, I've often wondered if the personality test largely measures how people interpret the questions.