I hope you're not training models based on the "which one is better" question, because that's incredibly subjective.

we considered which one adheres to the prompt more, which one has overall best aesthetics etc but ended up with a simple which one is overall better type question. it is easier for people to vote and decide one and still applicable as preference data at a larger scope (trading volume for simplicity).

the dataset is open source and we plan to train an aesthetics picker on it but obviously have to do proper evals (with at least 1M data) to come to a reasonable conclusion.

As feedback I can tell you it's not easy at all for me to decide which one is better. Maybe if the prompts would include the art style I would be able to clearly identify the better ones, but they don't. Style is where I see most of the differences.

Disclaimer: I only clicked on Surprise me.

comparing it to lmsys chatbot arena, what sort of an option would you expect? the prompts essentially come from public HF datasets like parti prompts where they test a bunch of stuff (prompt adherence, attention mapping [something in front of something else etc], aesthetics, photo-realism, etc.) so it is hard to ask about each category.

The question is ok but I need to have a clear input for me to decide which one is better. For example: A serene forest night, a lamp-lit path leads to a cozy wooden house. It comes up with a very detailed almost photorealistic image of the scene, while also bringing up a very well painted one. What do I choose? The input didn't mention anything about the style so it's very hard for me to pick a winner unless (like I said) I'm incredibly subjective.

i see about that case, and yeah you are right. we probably need realistic/artistic tags as you mentioned. thanks for the example! we'll probably include something like that in the next release and group models by ELO on different categories (can be considered like language analogue)

Glad I could help!