News for hackers. Stuff that matters.

A comment on Marcus AI Claims Dataset. Parent comment

Re:Marcus AI Claims Dataset by dvt on Tuesday March 03, 2026 @09:13PM (#47242131)
Gotcha', but I'm just trying to see the audit for how claim X was rated and based on what sources. If we're looking at the Claude logs, we have huge files that have things like this[1]:

    {"id": "claim_0081", "date": "2023-02-11", "claim": "Current Level 2 self-driving operates under easy conditions and is nowhere close to handling real-world complexity.", "type": "descriptive", "target": "Level 2 self-driving", "status": "supported", "horizon": null}
Why is this supported? How is this supported? Waymo would probably disagree, etc. Here's another one:

    {"id": "claim_0083", "date": "2023-02-11", "claim": "Tesla's product naming ('Autopilot', 'Full Self Driving') misleads customers into thinking the cars are more capable than they are, potentially causing accidents and deaths.", "type": "causal", "target": "Tesla marketing", "status": "supported", "horizon": null}
I fully agree that TSLA engages in all kinds of deceptive marketing, but to fully support the stunning claim that it potentially causes deaths is, uh, a bit much. I mean, at least tell me who's saying this. What's the provenance?

If Claude itself rated the claims, which seems the be the case unless I'm totally off base, I fail to see how we're actually doing anything at all here. Right now I'm working on a local research agent, and I'm being absolutely meticulous about storing browsed webpages, snippets, etc. into short-term (session) LLM memory or a long-term (cross-session) SQLite db.

[1] https://github.com/davegoldblatt/marcus-claims-dataset/blob/...

[ Parent | on HN ]
Re:Marcus AI Claims Dataset by davegoldblatt on Tuesday March 03, 2026 @09:29PM (#47242245)
You're not off base. There's no per-claim evidence trail. The Claude pipeline scored claims using model judgment and stored verdicts without the reasoning or sources behind them. Your Tesla example is a good one. There is real evidence for that causal claim (NHTSA investigations, documented crashes where Autopilot naming was cited as a factor), but the dataset doesn't show you any of it. Right now it's just "Claude said so." What the dataset is currently good for: extraction, clustering, timeline structure, and surfacing drift patterns across claim families. When 120 hallucination claims come back 57% supported and 63 bubble claims come back 41% contradicted, the distribution tells you something even if any single verdict is arguable. What it's not good for: claim-by-claim adjudication auditability. You're right that without provenance, this is closer to a structured first draft than a final verdict. Your approach of storing browsed pages and snippets per-claim is what auditable looks like.
[ Parent | on HN ]