Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction
by jauws on Tuesday February 03, 2026 @03:25PM
(#46876782)
Thanks for the feedback. What would you need to see to change your mind?
[ Parent
| on HN ]
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction
by mp_mn on Tuesday February 03, 2026 @03:33PM
(#46876879)
There's more quality fiction out there than you or I will ever have time to read. I don't see a purpose in flooding the world with more mediocre to unreadable fiction.
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction
by jauws on Tuesday February 03, 2026 @04:13PM
(#46877376)
Realistically, I don't think anyone will be spending hours here instead of reading real fiction anytime soon (I personally wouldn't). There's just so much nuanced complexity when it comes to creative writing as a domain (long-form outputs, creativity, etc.) that coming up with better annotation methods has massive applications in other research, like in scientific discovery. "AI Wattpad" just happens to be a convenient form factor for crowdsourcing from an HCI perspective. I hope you give it a chance.
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction
by mp_mn on Tuesday February 03, 2026 @04:41PM
(#46877712)
OK, so you already recognize these stories aren't something that people are going to spend time sorting through. How could you possibly then get any usable preference data out of this?
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction
by jauws on Tuesday February 03, 2026 @04:53PM
(#46877855)
If you look at similar live benchmarks like LMArena or Design Arena, there's an extremely large number of unique annotators, with a low number of annotations per person - which is normal. However, since this platform is designed to generate fiction catered to individual interests, my hypothesis is that it'll be an added boost of novelty that will help aggregate enough usable data over time.
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction
by mp_mn on Tuesday February 03, 2026 @05:06PM
(#46878005)
I tried reading the two top rated stories. They're both unreadable gunk. Why would I (or anyone else) go for another? Why would I tell anyone I know to spend their time reading this?
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction
by empath75 on Tuesday February 03, 2026 @04:03PM
(#46877259)
I am not going to argue this on the basis of LLM's suck at fiction, because even if it's true, it's not really that relevant. The problem is that what LLM's are good at is producing mediocre fiction particular to the tastes of the individual reading at. What people will keep reading is fiction that an LLM is writing because they personally asked it to write it.
I don't want to read fiction generated from someone else's ideas. I want to read LLM fiction generated from my weird quirks and personal taste.