News for hackers. Stuff that matters.

jauws

Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction on Tuesday February 03, 2026 @08:20PM (#46880093)
Definitely on the to-do list! Right now, there's smth called fork (inspired by Github fork), where it lets you remix the story with a given input. It might be cool for you to mess around with.
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction on Tuesday February 03, 2026 @08:18PM (#46880074)
Thanks Josh! I tried GEPA previously back when it was still 1-shot generation. It actually ended up working really well for some models and horrible for others, so I decided to scrap for a more generic prompt instead to make the benchmark a bit more rigourous.
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction on Tuesday February 03, 2026 @04:53PM (#46877855)
If you look at similar live benchmarks like LMArena or Design Arena, there's an extremely large number of unique annotators, with a low number of annotations per person - which is normal. However, since this platform is designed to generate fiction catered to individual interests, my hypothesis is that it'll be an added boost of novelty that will help aggregate enough usable data over time.
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction on Tuesday February 03, 2026 @04:38PM (#46877661)
I hope it's clear that the stories aren't being generated one-shot. I'm sure there are flaws that I haven't perfectly accounted for in the agent-loop, but because we randomize the models for each of the brainstorming -> writing -> memory parts, bad intermediate outputs will affect the final output as well. That's why unless we have above average models across all 3 stages, it might be worse than what you're used to. It's a trade-off to get more granular results. Hope you can give it a chance.
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction on Tuesday February 03, 2026 @04:13PM (#46877376)
Realistically, I don't think anyone will be spending hours here instead of reading real fiction anytime soon (I personally wouldn't). There's just so much nuanced complexity when it comes to creative writing as a domain (long-form outputs, creativity, etc.) that coming up with better annotation methods has massive applications in other research, like in scientific discovery. "AI Wattpad" just happens to be a convenient form factor for crowdsourcing from an HCI perspective. I hope you give it a chance.
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction on Tuesday February 03, 2026 @03:29PM (#46876837)
Happy to engage if you have concrete criticisms.
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction on Tuesday February 03, 2026 @03:29PM (#46876833)
If you have specific objections, I’m open to hearing them.
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction on Tuesday February 03, 2026 @03:28PM (#46876822)
Would love to chat! Here's my email: [email protected]
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction on Tuesday February 03, 2026 @03:25PM (#46876782)
Thanks for the feedback. What would you need to see to change your mind?
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction on Tuesday February 03, 2026 @03:23PM (#46876758)
Thanks for the feedback - looking at the rest of the comments, I definitely agree it seems to be a common theme. Will do better to fix those issues so there's less noise.
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction on Tuesday February 03, 2026 @03:19PM (#46876695)
I think there's interesting work to be built on this data beyond just generating and sorting slop. I didn't build this because I enjoy having people read bad fiction. I built it because existing benchmarks for creative writing are genuinely bad and often measure the wrong things. The goal isn't to ask users to read low-quality output for its own sake. It's to collect real reader-side signal for a category where automated evaluation has repeatedly failed.

More broadly, crowdsourced data where human inputs are fundamentally diverse lets us study problems that static benchmarks can't touch. The recent "Artificial Hivemind" paper (Jiang et al., NeurIPS 2025 Best Paper) showed that LLMs exhibit striking mode collapse on open-ended tasks, both within models and across model families, and that current reward models are poorly calibrated to diverse human preferences. Fiction at scale is exactly the kind of data you need to diagnose and measure this. You can see where models converge on the same tropes, whether "creative" behavior actually persists or collapses into the same patterns, and how novelty degrades over time. That signal matters well beyond fiction, including domains like scientific research where convergence versus originality really matters.

Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction on Tuesday February 03, 2026 @03:02PM (#46876451)
There's 151 models there right now (with all the latest Anthropic models), it's all randomized, it's just that there aren't enough annotations for the anthropic models to be elicited right now.
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction on Tuesday February 03, 2026 @02:00PM (#46875561)
Thanks for letting me know - the UI issues are definitely on me (fixing asap). Feel free to generate a story or two - right now there's not enough annotations to make "top-rated" a valid moniker.
Re:Show HN: I built "AI Wattpad" to eval LLMs on fiction on Tuesday February 03, 2026 @01:50PM (#46875419)
Ah shoot - thanks for letting me know. I'm still a noob on frontend so still learning as I go.

Show HN: I built "AI Wattpad" to eval LLMs on fiction

https://narrator.sh/llm-leaderboard

I've been a webfiction reader for years (too many hours on Royal Road), and I kept running into the same question: which LLMs actually write fiction that people want to keep reading? That's why I built Narrator (https://narrator.sh/llm-leaderboard) – a platform where LLMs generate serialized fiction and get ranked by real reader engagement.Turns out this is surprisingly hard to answer....

( Read More... | 34 comments | on HN )