[dead]
Gotcha', but I'm just trying to see the audit for how claim X was rated and based on what sources. If we're looking at the Claude logs, we have huge files that have things like this[1]:
{"id": "claim_0081", "date": "2023-02-11", "claim": "Current Level 2 self-driving operates under easy conditions and is nowhere close to handling real-world complexity.", "type": "descriptive", "target": "Level 2 self-driving", "status": "supported", "horizon": null}
{"id": "claim_0083", "date": "2023-02-11", "claim": "Tesla's product naming ('Autopilot', 'Full Self Driving') misleads customers into thinking the cars are more capable than they are, potentially causing accidents and deaths.", "type": "causal", "target": "Tesla marketing", "status": "supported", "horizon": null}
If Claude itself rated the claims, which seems the be the case unless I'm totally off base, I fail to see how we're actually doing anything at all here. Right now I'm working on a local research agent, and I'm being absolutely meticulous about storing browsed webpages, snippets, etc. into short-term (session) LLM memory or a long-term (cross-session) SQLite db.
[1] https://github.com/davegoldblatt/marcus-claims-dataset/blob/...
Re:Marcus AI Claims Dataset
by davegoldblatt on Tuesday March 03, 2026 @09:29PM
(#47242245)
You're not off base. There's no per-claim evidence trail. The Claude pipeline scored claims using model judgment and stored verdicts without the reasoning or sources behind them.
Your Tesla example is a good one. There is real evidence for that causal claim (NHTSA investigations, documented crashes where Autopilot naming was cited as a factor), but the dataset doesn't show you any of it. Right now it's just "Claude said so."
What the dataset is currently good for: extraction, clustering, timeline structure, and surfacing drift patterns across claim families. When 120 hallucination claims come back 57% supported and 63 bubble claims come back 41% contradicted, the distribution tells you something even if any single verdict is arguable.
What it's not good for: claim-by-claim adjudication auditability. You're right that without provenance, this is closer to a structured first draft than a final verdict. Your approach of storing browsed pages and snippets per-claim is what auditable looks like.
Small aside: I'm only bringing this up because last year I worked on a game where you had to solve various moral dilemmas in a 1v1 situation (think trolley experiment and one player says "flip the switch" and the other says "don't flip the switch")—the idea was to get an LLM to rate the arguments in a fun turn-based online game. I built it out, but I kind of gave up when I realize how absolutely awful the LLM was at actually rating arguments and their nuances. Who won legitimately felt more like rolling a dice than a verdict given by a real judge or a philosophy professor grading a paper. I put that project aside, but might do a Show HN at some point since the game is basically done.
Adjudication[1]—which is the real meat of this project—is done in a very partial way and I genuniely see basically zero value. Why not crawl reddit (or HN)? I know that also has issues, but it at least has more variety of tone.
[1] https://github.com/davegoldblatt/marcus-claims-dataset/blob/...