Yes, but I think it is still a viable metric to some degree. I wondered about Gemini being dead center here. At first it was obvious that it was actively trained to give biased responses to anything controversial. It was deservedly made fun off because it tried to warp reality. I still don't trust it today, although that is pretty much true for any model.

I don't think that craziness was baked into the model, it was in the prompt they used. So, easy to fix once they realized how stupid it was.

The alternative is a High-Dimensional / Embedding-Like Approach where question responses aren't tied to fixed axes, but rather the full response set is treated as a point in latent space.

Then it's on the researcher to examine the clusters and assign labels. There's also not a nice mapping that's a-priori interpretable in low-dimensional pre-existing axes.

Probably only used in research than consumer websites, under more controlled conditions; there are very few public political tests doing this transparently

Exactly. These models don’t hold coherent views. You can prime any of them to agree with any view.

Which is good as it is bad.

Me: "Please make an app that does X in C"

LLM: "C sucks donkey balls, use Rust instead".

It's hard to have a general purpose tool that both has and does not have opinions.

Your multimeter reads out voltage relative to the black terminal, it's your responsibility to find the ground plane.

My multimeter doesn't need me to tell it how much a volt is or feed it subjective measurements of what resistance means

Resistance is when you're nobly standing up to the other side and things get a little out of hand. Domestic terror is when the roles are reversed.

I used voltage as an analogy as it represents potential energy, which is relative. Political stance is a potential energy; action (political or otherwise) is, of course, an actual flow of energy. Resistance (via my analogy) is a potential difference across a load, and can be measured as how much energy is needed versus the action you get (R = V/I); via your analogy, it may be how strong a political action you get for a given delta in political stance.

So far there is no moral dimension to any of this. I think that is correct - the morality of a political stance depends on the member of the polity and the outcome that stance will have for them (which breaks down into perceived and actual outcomes). A functioning political system (the thing that governs political stance and political action) will ensure the balance of actual outcomes benefits the whole of the polity.

The delta between Resistance and Domestic Terror is then a question of whether the actors are party to the political system. (Back to the circuit analogy: domestic terror is the massive bulk capacitor charging off the energy in the circuit but with no defined discharge path in the system - until it hits a charge limit, short-circuits, and discharges.)

The convention for what a volt and ohm is has reached a steady state consensus, so the political debate there is minimal, and multimeters of the world can stand united.

OTOH, your laser distance meter may need you to set it in metric or imperial.

Fair; have tried to combat this issue in a few ways.

Each model's position is scored against outside political-science data (Chapel Hill Expert Survey for party positions, World Values Survey for where populations sit).

The stance coding is done by a separate model with a published prompt + a second model from a different lab re-scores a sample and we publish where the two disagree.

So not perfect but (as far as I can tell) one of the more defensible approaches.

Isn't it completely subjective to the point in time Overton window?

Yes, there's no way to avoid this really. You have to pick some point on the graph as the origin.

I’d argue the entire concept of being “unbiased” is flawed. All humans are biased all of the time, bias towards “the centre” is still a form of bias.

One cannot escape ideology, one merely swaps one ideology for another

This is especially apparent in the 'worldview' sorting under the `bias` section, which lists the German FDP to be further right than the CDU (which is nonsense) and also barely registers the FDP as libertarian when they are a free speech, small government, personal responsibility and free market party. They also register "Die Linke" as Libertarian-Left, which could not be further from the truth. "Die Linke" barely has libertarian values at all, being pro state-governed economy, having an ultimate goal of democratic socialism and they're certainly big government. They're also leading a large deposession effort for large landlord companies. I'd honestly put them into "Auth-Left" territory.

So yeah. The bias is a bit nuts and you could reasonably accuse the study/report of misdirection/misinformation and plain fasehoods.

Yep. All studies like this are just measuring "how much does the model agree with my preconceived notions?"

The other issue is that it is going to depend very strongly on how you ask.

They have a self-test you can take to compare yourself to the models, and one of the questions ends in “…even if some economists warn about bad outcomes from this”

That’s a crazy bias to throw into a question. Especially because it’s a relatively contested topic, from an economics research perspective.

How is adding that some economists warn about negative outcomes being biased when your comment indicates that indeed, some economists do warn about negative outcomes (ie, “some…negative” is what “contested” means)?

It's not a question of whether it's biased to say that, rather whether it would tend to bias the model's response (e.g. toward weighting the stated downsides as more imported than the implicit upsides). But that said, choosing to highlight particular facts while leaving others silent is a very common sign of bias.

Presumably some experts are also optimistic about positive benefits, so we already have bias from only focusing on the negative.

And, in fact, this is true of every single political question you can ask: the 10th dentist is concerned about negative outcomes of brushing your teeth daily. So there's a bias in only saying this in some cases, but not others.

and the investigator vs the anonymous collective compressed version of whatever is in question.

Abortion comes to mind for myself. I would probably be considered rather conservative but to me any abortion law is absolutely insane. If I got a woman pregnant though I would want to have the child.

So am I for or against abortion?

To me, all modern "politics" is really just the emotional reaction to a slogan and bumper sticker version of a complex issue.